Problems with the notifications

jwielsch · August 17, 2020, 6:36am

hi folks,

i have a problem with the notifications:

The users can not clear all their notifications with the new event store - but I can.

Any Idea where to start?

The only thing I have found in the logs after setting Trace on org.xwiki.notifications is below.

The information in second row is not always the same.

org.xwiki.eventstream.EventStreamException: Unsupported operator [DOCUMENT_VERSION ENDS WITH ".1"]

org.xwiki.eventstream.EventStreamException: Unsupported operator [SPACE STARTS WITH "Informationstechnologie"]

the whole thing:

2020-08-17 08:13:51,385 [Notification event executor: list : 5ALERT/20xwiki:XWiki.JWielsch/5/false/1949696012] DEBUG o.x.n.s.i.EventSearcher        - Failed to get events from the EventStore. Trying on the legacy store
org.xwiki.eventstream.EventStreamException: Unsupported operator [SPACE STARTS WITH "Informationstechnologie"]
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:203)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:182)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:182)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:190)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:181)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBinaryOperator(ExpressionNodeToEventQueryConverter.java:181)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:115)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseOtherOperation(ExpressionNodeToEventQueryConverter.java:275)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parseBlock(ExpressionNodeToEventQueryConverter.java:117)
    at org.xwiki.notifications.sources.internal.ExpressionNodeToEventQueryConverter.parse(ExpressionNodeToEventQueryConverter.java:105)
    at org.xwiki.notifications.sources.internal.EventQueryGenerator.generateQuery(EventQueryGenerator.java:57)
    at org.xwiki.notifications.sources.internal.EventSearcher.searchStoreEvents(EventSearcher.java:106)
    at org.xwiki.notifications.sources.internal.EventSearcher.searchEvents(EventSearcher.java:84)
    at org.xwiki.notifications.sources.internal.DefaultParametrizedNotificationManager.getEvents(DefaultParametrizedNotificationManager.java:126)
    at org.xwiki.notifications.sources.internal.DefaultParametrizedNotificationManager.getEvents(DefaultParametrizedNotificationManager.java:111)
    at org.xwiki.notifications.rest.internal.DefaultNotificationsResource.getCompositeEvents(DefaultNotificationsResource.java:177)
    at org.xwiki.notifications.rest.internal.DefaultNotificationsResource.lambda$getCompositeEvents$0(DefaultNotificationsResource.java:167)
    at org.xwiki.notifications.rest.internal.NotificationEventExecutor$CallableEntry.execute(NotificationEventExecutor.java:194)
    at org.xwiki.notifications.rest.internal.NotificationEventExecutor$CallableEntry.call(NotificationEventExecutor.java:155)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

surli · August 17, 2020, 8:05am

Hi,

I just checked in the code quickly about:

org.xwiki.eventstream.EventStreamException: Unsupported operator [SPACE STARTS WITH "Informationstechnologie"]

apparently it’s because some operators are not yet supported with the new event store (cf: https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwiki-platform-notifications/xwiki-platform-notifications-sources/src/main/java/org/xwiki/notifications/sources/internal/ExpressionNodeToEventQueryConverter.java#L202). I advise you to open a JIRA issue about it on https://jira.xwiki.org and switch back to the legacy store in the meantime as indicated in https://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/12.6/#HNeweventstoreandprefilteringbydefault

If you hit a blocker bug you can go back to:
- post filtering of event using the property notifications.eventPrefilteringEnabled in xwiki.properties
- use only the database store by disabling the new one using the property eventstream.store.enabled in xwiki.properties

To clarify are you using XWiki 12.6.1 or 12.6? It might be something fixed on 12.6.1 but could also be related to the stacktrace you found. In any way if you get the problem again after switching back don’t hesitate to open a JIRA issue. We’re actively working on notifications so it’s possible we introduced regressions.

Thanks

vmassol · August 17, 2020, 8:34am

If this is the case then we should not promote the new store by default (Why make it hard for users and offer a subpar experience), until it’s ready to be used.

cc @tmortagne

vmassol · August 17, 2020, 8:35am

12.6.1 (see the tag in the post).

vmassol · August 17, 2020, 8:44am

@jwielsch Could you show us the list of notification filters you have? We’d like to understand which one(s) is causing the trouble.

vmassol · August 17, 2020, 8:56am

Seems that @surli found it: https://matrix.to/#/!ikPtGZaGWtyblizzlR:matrix.xwiki.com/$1597653981774Xjzxv:matrix.xwiki.com?via=matrix.org&via=matrix.xwiki.com&via=dodsorf.as

jwielsch · August 17, 2020, 9:02am

I’ll send them as soon as I can. I am in meetings til 2pm.

surli · August 17, 2020, 9:24am

So after investigation I have some more information. We have actually a fallback mechanism in the new Notification system: the stacktrace you posted shows that the new store doesn’t properly handle a filter, but as specified in the message:

Failed to get events from the EventStore. Trying on the legacy store

it fallbacks on the legacy one. The error you obtain is informative only here and shouldn’t result in any regression on your side.
So it means:

you should still open a JIRA issue for us about it
it’s not mandatory for you to perform the changes I advised in my first answer. It’s not as harmful as I thought originally

Now regarding the “clear all notifications” original issue, I would need some more precisions about the problem: what happens exactly for your users? Do they keep having the same notifications appearing even after the clean? Or only some of them?
Note that AFAIK the new mechanism is entirely asynchronous, so it might be a timing issue (would be still a usability bug for us): do you know if the notifications are cleared after some time?

vmassol · August 17, 2020, 9:26am

BTW the message is a DEBUG one so it’s not a problem, as Simon indicated.

vmassol · August 17, 2020, 9:31am

So @surli has investigated and it seems we’re safe since:

The events are stored in the 2 stores (old and new)
When the new store fails, we fallback to the old store

In addition we missed the fact that the message was a DEBUG one and thus it’s not a problem.

jwielsch · August 17, 2020, 12:04pm

Thank you all very much for the fast respond. I tried to figure out why the others can’t but I can clear all notifications and this was the only hint. So I posted the message - as I thought it would be better than just saying “i have a problem”.

I switched back to the old store and now others can clear all notifications as expected.

Shall I create a jira for this? (I have no more information and no logs to show - just the fact.)

surli · August 17, 2020, 12:11pm

Yes definitely. Provide as much info as you can, especially if you have some custom rights configuration for your user, or if it’s related to some sub-wikis, etc. Anyway we will follow up on the JIRA.

jwielsch · August 18, 2020, 4:17am

Gabriela Anechitoaei was quicker than me in creating the jira.

https://jira.xwiki.org/browse/XWIKI-17655