Xwiki 10.5 entended cpu spike while adding new pages

Help / Discuss
,
#1

So I’ve a new problem,

When I am adding new pages to the wiki, I am getting extended cpu spikes where the cpu is pegged to 100% for a few minutes before xwiki responds and completes the request. While this is happening, the wiki becomes unresponsive.

I’ve done a thread dump while this is going on (attached -> dump.txt (1.9 MB). I’d be grateful of someone could have a look and see if there is anything obvious I should investigating or fixing.

The following is a jconsole screen dump where you can see the cpu spiking:
33

Remove all Custom Notification Filters/Filters for non existing pages
#2

I’ve not looked at your dump file yet but one idea is that when saving a page all listeners that listen to page changes execute. XWiki Standard has several of them and extensions and users can contribute new ones. By any chance would you have some custom listeners?

I’d check the listeners that execute and what they do.

#3

I have no custom listeners, only some added extensions from extensions.xwiki.org.

Should I try disabling all extensions temporarily and see if the problem goes away?

#4

hmm… I cannot disable extensions temporarily it seems.

Any other suggestion how I can find out where the cpu is spending its time?

#5

Typically the tool to analyze CPU issues is a Java Profiler (such as Yourkit). See https://dev.xwiki.org/xwiki/bin/view/Community/Profiling

#6

I’ll try that now.

In the meantime, java-melody is showing me this:
25%201

It seems notifications seem to be where excessive time is being spent. Does that look right to you?

edit: well, that pic is illegible. try this one instead: https://www.dropbox.com/s/ewx5exj8if5tkyd/xwikicpuspikers.png?dl=0

#7

Normally and AFAIU notifications code won’t execute when new pages are created. Notifications code execute when the user clicks on the alarm bell menu and when the scheduler sends notification emails.

Do you get CPU spikes even when no user is opening the notifications menu and when no notifications scheduler job is running?

#8

I don’t believe so.

I just clicked on the notification bell icon on the main wiki page and the cpu just spiked briefly. Also, I get the spinner icon and nothing appears under notifications.
49

#9

hmm… I don’t know if this is related, but under my notifications preferences for my user id, I have almost 730 filters defined. That seems a bit strange.

(I’m deleting them 1 by 1, but it’s slow as the cpu spikes every time I make an edit to them)

#10

This is probably related to https://jira.xwiki.org/browse/XWIKI-15371 and https://markmail.org/message/74zmkivpjdl5akad

#11

Any suggestions as to what I can do in the meantime to get the performance back? Is there a way to globally disable the notifications so this cpu hogging stops happening?

#12

and yes, I’m definitely seeing the queries described in the markmail.org mailing list post you linked to.

select activityev0_.ase_eventid as ase1_33_, activityev0_.ase_requestid as ase2_33_, activityev0_.ase_stream as ase3_33_, activityev0_.ase_date as ase4_33_, activityev0_.ase_priority as ase5_33_, activityev0_.ase_type as ase6_33_, activityev0_.ase_application as ase7_33_, activityev0_.ase_user as ase8_33_, activityev0_.ase_wiki as ase9_33_, activityev0_.ase_space as ase10_33_, activityev0_.ase_page as ase11_33_, activityev0_.ase_hidden as ase12_33_, activityev0_.ase_url as ase13_33_, activityev0_.ase_title as ase14_33_, activityev0_.ase_body as ase15_33_, activityev0_.ase_version as ase16_33_, activityev0_.ase_param1 as ase17_33_, activityev0_.ase_param2 as ase18_33_, activityev0_.ase_param3 as ase19_33_, activityev0_.ase_param4 as ase20_33_, activityev0_.ase_param5 as ase21_33_ from activitystream_events activityev0_ where (activityev0_.ase_type=? and activityev0_.ase_date>=? and (activityev0_.ase_wiki=? and activityev0_.ase_page=?
or activityev0_.ase_wiki=? and activityev0_.ase_page=?
or activityev0_.ase_type=? and activityev0_.ase_date>=?
and (activityev0_.ase_wiki=? and activityev0_.ase_page=?
or activityev0_.ase_wiki=? and activityev0_.ase_page=?

  • 746 lines deleted *

or activityev0_.ase_wiki=? and activityev0_.ase_page=?)
or activityev0_.ase_type=? and activityev0_.ase_date>=? and (activityev0_.ase_wiki=? and activityev0_.ase_page=?
or activityev0_.ase_wiki=? and activityev0_.ase_page=?

  • 746 lines deleted *

or activityev0_.ase_wiki=? and activityev0_.ase_page=?)
or activityev0_.ase_type=? and activityev0_.ase_date>=? and (activityev0_.ase_wiki=? and activityev0_.ase_page=?
or activityev0_.ase_wiki=? and activityev0_.ase_page=?

  • 746 lines deleted *

or activityev0_.ase_wiki=? and activityev0_.ase_page=?)) and (activityev0_.ase_type<>?
or activityev0_.ase_version like concat(’%’, ?) escape ‘!’) and activityev0_.ase_user<>? and activityev0_.ase_user<>? and (activityev0_.ase_eventid not in (select activityev1_.ases_eventid from activitystream_events_status activityev1_, activitystream_events activityev2_ where activityev1_.ases_eventid=activityev2_.ase_eventid and activityev1_.ases_eventid=activityev0_.ase_eventid and activityev1_.ases_entityid=? and activityev1_.ases_read=1)) and (activityev0_.ase_eventid not in (? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ?)) and activityev0_.ase_hidden<>1
order by activityev0_.ase_date DESC limit ?

#13

If your user had ~700 pages in his watch list, it generates ~700 corresponding filters for notifications. This can be the cause of your problem.

Unfortunately, we have no in-the-box solution to clean these filters, but if you wish, you could try to delete the XWiki.WatchListClass object of your user profile. If it is not enough, try to delete the XWiki.Notifications.Code.NotificationFilterPreferenceClass but I doubt it will be useful.

I hope it helps,
Guillaume

#14

Where are these filters stored? If they’re stored in the database, is there a query I can run to clean them out?

#15

They are stored as XObject on your profile page. Use the Object editor to see them. You need to be an Advanced User to perform this operation.

#16

I’ll give that a try. Right now, deleting some of the filters can take half an hour, each.

#17

Yes, I can delete the notifications from the object editor, but it takes just as long for it to do whatever it is it is doing.

Safer to stick with the normal notifications editor in that case.

#18

@pdwalker once you’ve finished deleting the notification filters, it’ll be interesting to see if that’s the problem of your CPU spikes. If it is, I still don’t understand what would be the relationship between Creating Pages and Notification code executing though. We’ll see. Let us know when you’re done. Also we need to fix the Notifications performance issue we’ve discovered recently when there are lots of filters (one thing we need to do is check why this is taking long from a DB point of view since a WHERE clause with right indexes shouldn’t take that long).

#19

659 notifications to go…

#20

You’re not doing this manually, are you? :slight_smile: That seems insane…

You should script the removal of xobjects I think. That should go much faster.