Hacking Linux Kernel Through testing

From WorkOutWiki2009

Proposer

Poornima Nayak and Sudhir Kumar. We have been working in Linux Kernel testing and Test development for more then 3 years. We have been contributing to Linux Test Project from last 3 years.

Purpose

Objective of this workout is to introduce one of the easier interface to become a kernel hacker to the students community . A tester is also a hacker in a open source culture as he applies various techniques to learn the product on test. Testing is one of the crucial need of any project. Many giants like IBM, Novell, Redhat & many other company's are bridging the testing needs of Linux OS. Still being a open source project with huge developers community there is still a need for more testing and test automation. The project Linux Test Project(LTP) is one of the projects which facilitates test development & maintenance of the automated test cases by anyone in the community.

In this workout we will be focusing on test development for subsystems like Power Management & Resource Management, understand these subsystems and enhance them in future(sorry lmited time for the workout!!).

Abstract

Hacking Linux Kernel through Testing

Pre-requisites

1. You should have a working knowledge of Linux

2. The minimum requirement is to have Fedora10 on your laptop, however better if you have Fedora12. If you do not want to reinstall your machine then you need to compile and boot your laptop into the latest stable kernel. For whatever case ensure that you have memory resource controller and cpu controller compiled in your kernel. An easy check is cat /proc/cgroups should show memory and cpu enabled. Since distros may not enable memory controller by default you can boot with additional parameter cgroup_enable=memory

The config options for enabling cpu and memory controller in the kernel:

CONFIG_GROUP_SCHED=y

CONFIG_FAIR_GROUP_SCHED=y

CONFIG_RT_GROUP_SCHED=y

CONFIG_CGROUP_SCHED=y

CONFIG_CGROUPS=y

CONFIG_CGROUP_DEBUG=y

CONFIG_CGROUP_NS=y

CONFIG_CGROUP_MEM_RES_CTLR=y

CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

Languages, tools to be known

C, Shell scripting, Python, Git/quilt/Make(optional)

Getting and compiling the code

Links to overall design/architecture

For a detailed info please see the links at the bottom.

Power Management: Linux processor power management has been on power managing the processor while it is active: cpufreq, which changes the processor frequency and/or voltage and manages the processor performance levels and power consumption based on processor load. Another dimension of processor power management is processor ‘idling’ power: 'cpuidle' a generic processor idle management framework in Linux kernel. Problems given in the 'Tasks' section will help to get better understanding of cpuidle & cpufreq framework.

Resource Management: Typically Unix systems provide a limited control over system resources which is per task basis. Therefore to control a particular job which has a bunch of processes rlimit may not be a suitable way. Control groups(cgroups) is a mechanism to group a set of tasks into a single group. Control groups provide an infrastructure for a controller to limit the amount of resources allocated to the control group. Thus a controller can control the system resource distribution among these groups and user space can build a policy on top of it. For further information on cgroups and controllers we highly recommend you to go through the documentation in the kernel source code.

Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. The LTP test suite is a collection of automated and semi-automated tests for testing various aspects of the Linux operating system. The dominant programming language in LTP is ANSI-C followed by Bash, Perl & Python. It has a huge collection of test cases for various subsystems of Linux Kernel.

Pre-event planning

Since the time for the workout is limited we expect you to do a little home work in addition to the above Pre-requisites. There is a "how-to" on the LTP home page which is a good reference for beginners. http://ltp.sourceforge.net/documentation/how-to/ltp.php

Download the October release of LTP from the link on the home page. You can also download from the mirror given below.

http://nchc.dl.sourceforge.net/project/ltp/LTP%20Source/ltp-20091031/ltp-full-20091031.tgz

Untar the tarball and quickly go through the INSTALL and README files and see if all the required packages are installed in your machine. Also try to run some small testcases. we suggest you not to run any of the stress or crashme tests on your laptop as they are meant to stress the system.

Tasks

Develop test cases for Power Management & Resource Management subsystems of Linux Kernel and understand those subsystems.

PM

1) Functional test CPU freq scaling governors

Objective:

Change various governors and verify the following attributes:

- Does the frequency change in the hardware, use workload to test

- Is the information available in sysfs cpufreq and /proc/cpuinfo reflect the running frequency

- In case of ondemand and conservative governors, does the frequency change closely follow workload utilisation

- Does the thresholds and parameter to these governors provide expected results

Requirements:

- Workload that can measure cpu's capacity like ebizzy or any other cpu intensive workload that is repeatable

- Workload that can load each CPU to a certain preset utilization level

Issues: - Frequency change in hardware may be linked with system topology like dual core, sibling threads etc.

2) Functional test CPU idle governor

Objective:

Change various governors and verify the following attributes:

- Does the C-State residency change based on idle interval

- Does the thresholds and parameter to these governors provide expected results

Requirements:

- Workload that can inject timers at specific rate on a given CPU. User space program with varying usleep() may help but will need hr timer framework to use sub milli second accuracies.

3) Functional testing of PMQoS

Objective:

Change PM QoS requirement and verify its effect on C-State residency

Requirements:

- Workload that can inject timers at specific rate on a given CPU

- Test program will issue different QoS requirement and verify its effect on C-State residency

4)Ensure system reliability by switching cpu online & offline while changing cpu governors in parallel

RM

1. Memory Controller supports hierarchy of groups. Write a test which validates the upper limit imposed by the topmost hierarchy.

2. Test the testcase no 1 for subgroups far below in hierarchy.

3. A group has n siblings. They start consuming memory and the total increases to more than parent's limit. Check if the group with max memory consumption gets it's task killed.

4. memory.memsw.limit_in_bytes puts limit on memory+swap usage. memory.limit_in_bytes put limit on memory usage.Write a test which uses lots of memory, reaches swap limits and ensure it gets killed.

5. A task is running in a group which has a limit N on memory.limit_in_bytes but no limit on memory.memsw.limit_in_bytes. The system has swap memory in use and the task's memory consumption is >> N. Therefore the task should be using a big amount of swap memory(say M) > N. Try to set the memory.memsw.limit_in_bytes to a low value say N. It should fail to change the limits.

6. In test no 5 instead of changing the mem+swap limits turn the swap off. Since the memory usage of the task is more than the current limits the task should be killed.

7. Suicide!

       - This is just for the fun and not for developing a test. Check the system's memory usage at present. Change memory.limit_in_butes to a value lower than memory usage. See how your system starts swapping in
and out the memory and becomes less responsive.

       - Now turn swap off. see the OOM getting invoked.

       - Or just put memory limits on your machine to few MB.

       - If you want to look back into your past(when you were proud of your laptop having 256 MB RAM), change the root groups' memory limit to 256 MB and see if your old laptop would have been able to fulfill  your
current requirements.

8. There are many more testcases but the time is limited. Here onwards we encourage you to identify scenarios where you want to test memory controller.

We will help you to integrate the developed test cases to LTP. Also a short session on LTP would be conducted if required. Every automated test case should have action as well as result verification code. An automated test case is not completely automated without result verification code. If time permits we would like to take you through some of the functions providing the functionality that you tested or developed tests for.

Existing work

We already have Power Management & Resource Management functionality verification test cases. But we would like to improve test coverage of these features in Linux Kernel as they are highly focussed features in Linux community recently. For the problems mentioned in 'Tasks' section there are reusable functions in ltp/testcases/kernel/power_management. When you develop code to for the above problems its a good practice to use the already defined functions.

Getting in touch

During the workout we will be hanging @ IRC - #foss-kernel @ OFTC

Later also, please feel free to be in touch with us @

LTP list <ltp-list@lists.sourceforge.net>
irc.freenode.org #ltp
LTP maintainer: subrata@linux.vnet.ibm.com. 
Contributers: mpnayak@linux.vnet.ibm.com, skumar@linux.vnet.ibm.com

Reference links

memory Controller

1. http://lkml.org/lkml/2006/10/30/51

2. Files Documentation/cgroups/memory.txt, Documentation/cgroups/memcg_test.txt in the kernel code. Download the tarball from kernel.org and read these files. Documentation/cgroups/cgroups.txt gives an idea on how the controllers work in general.