08:59 <goyda> Hi, I am new here. I am trying to figure out a problem that is itching me a bit. I am not sure if it can be solved by systemtap, but I will it a shoot anyway.

09:01 <goyda> I have 2 RHEL vms installed on 2 cloud instances which have exact hardware configurations and same OS version, glibc, OS patch level etc. I am running exact same binary on the both machines with same input load. On one of the vms, the process consume 20% more CPU than other

09:03 <goyda> I am wondering if there is any way to find out why one is consuming more CPU than other

09:07 <goyda> lscpu command output on both looks same except for couple of flags. The one where the CPU consumption is less has got 2 extra flags, arat and tsc_adjust. Will that make any difference?

11:01 <fche> goyda, hi

11:01 <fche> both those different cpu flags are timekeeping related, and that could have some impact though I'd be surprised at 20%

11:02 <fche> is it possible that the vms are running with different levels of spectre/meltdown protection?

11:38 <goyda> thanks. I did check that, both are not patched with meltdown/spectre. Also, I wrote a test program that did not do any system call. It basically incremented a integer in a loop. Ran the program as 'time a.out' both the vms. On one vm it takes more time than other consistently

11:57 <fche> goyda, is the time in the vms accurate? it's unlikely, but one could be running its clock slower than the other, making the computation seem faster

12:04 <goyda> Khaled, How do I check that?

12:07 <goyda> Will the top command also affected by timer? Because in one of the machines top always shows more CPU consumption for the same process than other vm

12:22 <fche> it's a long shot -- if only one process in the vm is affected, then I would not suspect this sort of thing

12:23 <fche> I might move on to other things like caching behaviour, which perhaps stap/perf stats can help with. Maybe one of the vms runs on a core that has a cache-hogging sibling

12:29 <goyda> Ok, the test program I wrote simply increments a integer, not sure if caching can cause any problem.

12:29 <goyda> #include<iostream> #include <unistd.h> void func() { volatile size_t i = 0; while(i < 10000) { ++i; } } int main() { for(int i = 0; i < 1000000; ++i) { func(); } }

12:30 <fche> volatile ... so the compiler is probably pushing it into ram rather than register, so yeah caching could play a role

12:30 <fche> if there were contention

12:31 <goyda> Looks interesting. Is there any way I can confirm it?

12:32 <goyda> Perf stat a.out does not seems to be printing too many things.

12:32 <fche> compare it running on the two machines

12:35 <goyda> Unfortunately it is printing not supported for everything. Looks something is not on these vms.

12:35 <goyda> [root@fastVM ~]# perf stat -d ./a.out Performance counter stats for './a.out': 18559.135247 task-clock (msec) # 1.000 CPUs utilized 29 context-switches # 0.002 K/sec 0 cpu-migrations # 0.000 K/sec 305 page-faults # 0.016 K/sec <not supported> cycles <not supported> instructions <not su

12:37 <goyda> [root@slowvm ~]# perf stat -d ./a.out Performance counter stats for './a.out': 21625.409915 task-clock (msec) # 1.000 CPUs utilized 25 context-switches # 0.001 K/sec 0 cpu-migrations # 0.000 K/sec 305 page-faults # 0.014 K/sec <not supported> cycles <not supported> instructions <not su

12:38 <goyda> Let me try to remove volatile and try again. Hopefully it does not get optimized out

12:38 <fche> hm can you reconfigure these vms for the hypervisor to pass through hardware perfcounters?

12:39 <fche> with kvm it's something like cpu-model=host-passthru

12:44 <goyda> thanks, Unfortunately I don't have the permission to reconfigure it :-(. I got the root permission for the vms, can I do something with that/

12:51 <goyda> Anything else I can check?

12:52 <fche> do you know what other workload is running on the hosts supporting these vms?

12:52 <fche> it isn't one host for both, is it?

12:57 <goyda> I am not sure about the workload on these hosts.

12:58 <goyda> No these vms are located in 2 different sites, so they are not same hosts

13:02 <fche> ok. yeah with the limited tooling available, not sure what else to suggest

13:02 <fche> I'd want to look at the hosts & kernel & the hypervisor configuration

13:05 <goyda> Do you want me to paste any command's output from both vms?

13:05 <fche> not sure you can get enough info. the /proc/cpuinfo part you already collected, but kvm can/will lie

13:06 <fche> that's why I would start -at the host-

13:06 <fche> a vm is a prison in more ways than one :)

13:10 <goyda> :-). Let me try to get some info on the hosts side( It is bit bureaucratic process to get these details, so fingers crossed :-( )

13:12 <goyda> One question on "cpu-model=host-passthru" option, will it affect the performance?

13:12 <goyda> or anything else of the vms?

13:12 <fche> I don't think so, most likely, except that the host cpu's perfctr facilities would be available to the guest

13:13 <fche> so you have at least a chance at checking up on those caching stats

13:24 <goyda> Ok, Let me give a shot then. Thanks a lot. It was nice talking to you guys

13:24 <fche> no problem, let us know what you find

20:22 <irker256> systemtap: juddin systemtap.git:refs/heads/jafeer/pr23074 * release-4.0-186-gb73d653 / dwflpp.cxx dwflpp.h loc2stap.cxx scripts/dump-parameter-ref.sh tapsets.cxx: Some code clean up http://tinyurl.com/y5kg67ed