Brainstorming about XWiki Discussions

Development
#1

Hi everyone,

as part of NGI#2 we made the following proposal for Usecase 2:

Ability to provide discussions inside XWiki, and more specifically the ability to reply to activities. XWiki already provides Notifications but it’s currently not possible for users to discuss around them between themselves. The idea is to add a social aspect around XWiki activities. It also means that from other software it would be possible to know about shared documents and start a discussion around them.

This usecase is actually complex and is talking about different things, so in order to properly implement it both in ActivityPub extension, but inline with XWiki Standard wanted evolutions, we’d like to clarify it and ensure we’re all on the same page.

This brainstorming won’t necessarily go into all details for now, but we’re trying to take first the big decisions.

We see 3 important points in this usecase:

  1. it’s about providing discussions in XWiki (first sentence)
  2. the usecase is mentioning Notifications as a possible entry point for such discussions (second sentence)
  3. the usecase is mentioning the capability to start or contribute to a discussion outside (but related to) XWiki, but that would be somehow displayed also in XWiki

Definition of a discussion

Our understanding is then that the main goal is to be able to have discussions but we need to clarify what is exactly a discussion. The first vision is that a discussion is a thread of messages having a common topic.

Two things here:

  1. this common topic might be various things: it could be a message itself, but it could also be any kind of event in a wiki (a page edit, a new comment, a like, a mention, etc)
  2. a message is necessarily a user-written message for the specific thread: it means that we wouldn’t include in a discussion other events: you cannot have in a discussion around an edit event of a page, another edit event of the page for example

So the first question of this brainstorming is: do you agree with this definition?

Need of a messaging application

Following this definition, we envision that we do need a dedicated messaging application in XWiki to manage those discussions. This dedicated messaging application would have then several roles: to be able to display all discussions the user have been involved to (or all public discussions if any); to be able to participate to a discussion by answering to it and to be able to start a discussion by sending a first message.

Note that both the contributions to the discussions coming from XWiki and from other software (through ActivityPub), would be then aggregated in the same place.

We can imagine two situations without the messaging applications:

  1. all discussions taking place in Notifications
  2. all discussions taking place in comments

Situation 1 is not good for several reasons, one of them is the ability to properly display long threads of discussions in Notifications area. Second reason is the fact that Notifications are volatile (they are not aimed to be kept forever for users), while discussions should be persistent.

Situation 2 is not good either, simply because not all discussions would concern pages: how about starting a discussions from a message to another user? How about starting a discussion from a page deletion? etc

So the second question of this brainstorming is: do you agree that we need a dedicated messaging application to allow displaying discussions and maybe participate to them?

Note that for now, I’m not mentioning the Message Stream application, we might or not reuse it depending on the more specific usecase. We want first to know if we all agree that a Messaging application is needed here for the discussions.

Need of Notifications quick actions

As we mentioned in our definition of discussions, a discussion is starting from any kind of event: either a message sent to another user, or an edit event on a page, a mention made in comment etc.

All those events, by definition, appear first in the Notification Area (under the bell). So it seems a good idea for usability reason to be able to introduce quick actions in the Notifications Area, and the capability to start a discussion from an event.

I’m talking here about quick actions, plural, because we’re thinking about a generic mechanism not only useful for the discussions, but also for other usecases: you could imagine in the future, liking a notification, sharing it, etc, so it looks like the “start discussion” usecase would be only a first usecase of a more generic mechanism to interact with an existing notification.

Do you agree that such mechanism would be useful for XWiki Standard, and that it would be needed for properly using discussions?

Data representation of Discussions

This topic is a bit less advanced than the others, and we would need more a brainstorming than an agreement on it.

Basically the question is: how should we store the discussions data?
Personally I (Simon), was thinking that it would be easy to store them directly as specific kind of Event (in the Event store then), but after a migration to allow linking events to each other. Basically each Event Message would be able to make a link to the previous Event it would answer to, and the whole chain would constitute the discussion (the thread).

The advantage of this solution IMO is that it would benefit from the already existing work on event store and it would be as scalable as it.
The problems I see, besides the need of a migration potentially expensive, is that the capability to export easily users’ data: it would take some work to be able to export all discussions you’ve been involving to. Moreover, it also mixes two different completely different usecases in the same store, which is rarely good.

We think we can already discard the solution of storing discussions as xobjects, for the same reasons we think it’s a bad idea to only consider discussion messages as comments.

WDYT?

Simon and @mleduc

#2

A note about this. Actually this is currently a major problem with comments. For example we see on xwiki.org a lot of cases where users make comment on a doc page mentioning what would need to be improved to the page or some bugs, etc. These are transient because as soon as the recommended change has been integrated to the page or as soon as the bug has been fixed, we’d like to mark these comments as resolved and remove/hide them.

So I do think that we need the ability to mark a comment as resolved and possibly remove them.

#3

I’m not sure I understand this part. Why couldn’t we have a discussion around an edit of a page? That generates a notification (and a diff) and I’d find it very useful and good to be able to comment about that.

#4

Note that there’s a solution for this, which is to store comments in a different place than the page itself (it could be in another page dedicated to discussions/comments for the page or entity, or it could be in a different store). In any case, we already know that comments don’t scale so it might a good thing to do.

Related issues:

#5

I think I’d like that whatever solution is chosen, that the comment feature uses it. So it’s either about improving comments to make it more generic or to develop another feature and reimplement comments with it.

WDYT?

#6

We need to not forget to evaluate the message stream application: https://extensions.xwiki.org/xwiki/bin/view/Extension/Message%20Stream%20Application/

Also, related:

EDIT: Just seen that you mentioned the message stream application:

ok. I agree we need one.

#7

We shouldn’t forget starting with a page comment too as a use case.

(brainstorming mode)
Wild idea:

  • Consider everything in XWiki to be an Entity (a doc is an entity for ex, so is a wiki, a space, an xobject, an xproperty). We could imagine making a notification an entity too.
  • Then we can imagine supporting actions for all Entities, as we do already for docs: a URL for them, view them, delete them, comment on them, etc.
  • From a technical POV, the store used can be defined by Entity type. Notifications Entities would use the notifications solr store. The doc Entities use the doc store.

The advantage is to make it all more consistent with a single similar UI and ways to interact with any entity. It also gives a UI to discuss (comment) about any entity.
(/brainstorming mode)

#8

I’m not sure I would discard it too quickly. I think it can make a lot of sense. What we need to avoid for scalability reasons is to store many xobjects into a single page since that doesn’t scale currently (but we need to make it scale: http://jira.xwiki.org/browse/XWIKI-2874).

Could you elaborate about " for the same reasons we think it’s a bad idea to only consider discussion messages as comments"?

#9

Thanks for the note, that would be interesting to have the feature even if I’m not entirely sure that would make them as volatile as Notifications (at least for now). But anyway it’s part of another discussion :slight_smile:

Ok I have not been clear enough here and it’s an important point. I’m not saying that we couldn’t have a discussion around an edit of a page, on the contrary as I stated any discussion could have any event as starting point. So basically you could discuss around any event.

Now here what I’m saying is that if a discussion starts from an event, later on in the discussion a message won’t be any kind of event. It will be necessarily a user-written message (could be stored as an event though).
So to give you an example: you start a discussion around the Sandbox edition 1.1, later on there a Sandbox edition 2.1 event, this event won’t be included in the discussion. You could start a new discussion with it, but it will be another discussion.

I propose this definition, in order to scope a discussion as users’ messages around a same topic: including in those discussions multiple other events might maybe satisfy some usecases, but could also introduce a big mess in the discussions IMO.

I’m not sure storing the comments in another place would be enough. Here we would need to store them at another place, but we’d also need the capability in the UI to allow commenting anything and view the comments of anything.
Maybe it’s something we want in the future but we’re far from there yet.

That would be indeed an answer to my previous comment. If we consider that those discussions could be seen as a bigger implementation of comments, then yeah on the long term view we could imagine reimplementing comments with it. I would say that we need to take this into consideration for designing the discussion application, so that as most as possible we consider the future capability to migrate comments in them. Now I don’t think this should be a blocker if there’s some features of comments we don’t manage to properly design in the discussion app yet: this migration should be considered as a long-term work only, where the discussion app is more a short-term work.

So AFAIU commenting a page is a discussion start for you? It’s inline with the previous remark on comments, now it’s not something we considered when writing this. What’s interesting here is that it would raise some ambiguities about the event starting point of the discussion.

My proposal to start a discussion from an event is that the discussion is non-ambiguous about the state of the document when you start the discussion. If I take my example, if you start a discussion around edit of Sandbox 1.1 you’re commenting/discussion this version specifically.
Now, if you start commenting a page, you’re actually commenting what you’re seeing. But there’s various problems:

  1. right now, even if the page changed we keep displaying the comments which might not be relevant anymore to the current version (hence your remark before)
  2. if you want to post a comment attached to the event of the current edition of a page, it means being able to retrieve that event, and maybe it could be misleading: we could imagine that you want to start a discussion about the wiki content of the page, and the edition event of latest edition is actually an object edition. In which case the discussion wouldn’t be related with the diff of the event.

I agree that might be interesting, but we have to be careful also about creating Comment entity too then, to be able to comment on comment and create threads. Then commenting any entity wouldn’t necessarily use the store of entity commented, but it would use the store of Comment entity.
Now once again we have to be a bit careful with doing things by step: I’m not sure we can start with doing such kind of work for Discussion app if we need it in few months at best.

So the reasons about discarding comment were about the scalability problem, and the capability to discuss around any entity and not just page. Now I agree that if we refactor comments to be discussions we could consider them to be the same thing.

#10

A more general question I was asking myself is, do we want to promote chat-like “real-time” discussion or forum-like discussion based on longer messages?

A chat-like system will probably not be hierarchical, the UI will not be paginated and will be expected to be real-time (websocket…), and users expected to send mostly short content.
A forum-like discussion could be paginated, possibly made hierarchical, and users are expected to send longer content.

Messaging Application

I like the idea of making the messaging application generic in order to allow discussions to be related to different and possibly unanticipated kind of entities.

Regardless, I think the main features of the messaging application are:

  • Provides an API to send, read and update messages
  • Provides the main UI to give access to all the messages of a user
  • Provides generic UI components that can be reused to integrate messages on other parts of the wiki:
    • ActivityPub
    • Comments / discussions around a specific page version…
    • Annotations
    • Group discussions
    • User to User discussions
    • Quick answer/start a discussion from the notifications

Messaging Application Backend

Event Store

Pros:

  • Already existing and usable
  • Message Stream is already based on message stream and while its scope it more limited than what we want to do, it can be extended.

Cons:

  • Impact of a possibly large amount of messages on the performance?
  • The event store datastructure must be updated to allow events to be chained to their predecessors, which wouldn’t be needed if it wasn’t for the messaging application

xclasses/xobjects

It is already existing, well tested, and allows us to benefit fully from existing XWiki APIs.
If having a page for each message is not an issue, then this is probably a nice solution.
My main concern, but I don’t have much hindsight on the question, is about the possibility of making complex requests on xobjects scattered among many pages.

A new dedicated store

Pros:

  • Freedom to choose a datastructure and/or database engine that matches perfectly the needs of the messaging application.

Cons:

  • Much more work.
  • New store to maintain.

Message Stream application

The message stream application is based on the event store for its backend.
It is shipped with XS by but is deactivated by default.
It provides a macro that allows sending a message publicly, to the user’s profile page, to another user profile page, or to a group.
It does not provide a main UI to consult all the send or receive messages.
It does not provide a way to continue a discussion by answering to a received message.

Roadmap

The roadmap below is a draft based on my understanding and the needs of the development of ActivityPub, and should be updated based on the stakeholder’s feedbacks.

Features needed in priority in order to allow the integration with ActivityPub:

  • Providing the messaging API and its backend
  • Providing the main messaging UI

Later:

  • quickly answer to a message from the notifications
  • integrating and migrating the comments and annotations to the messaging application
  • integration of the messaging application for new usage
  • starting a discussion from a mention
#11

IMO what we are discussing now is about what you call a forum-like discussions. The difference I see with a chat is that in a chat discussions are generally perform in sync and are much more volatile, while in a forum-like discussions everything is async so the messages are always persistent.

We might need a chat later in XWiki, maybe integrated in the same UI, but it would be another topic.

IMO the main UI should allow to get access to the messages:

  • written by the current user
  • where the current user is mentioned
  • concerning a page watched by the current user
  • that are flagged as public

We might need different views for this, but it’s important to take right now into account that this UI won’t display only the discussions a user has been involved on.

I don’t really see how a large amount of messages could impact the event infrastructure since it would be always less than the number of events. So I don’t think it would be that much a problem.

Even without considering a page for each message, we might consider a page per discussion. That would probably make sense and that might scale.

The fact that xobjects are scattered among pages is not a problem since they’re all stored in the same table in the DB so it’s easy to perform complex queries with them most of the time.
Now as I said above, we could even consider xobjects related to the same discussion being stored in the same page.

I would be -1 against a new kind of store unless we have really good reason for it.
Now maybe you meant a new Solr core for storing the data, like we use for Ratings?

Thanks for the sum up, so what do you propose? To take it back and refactor it to add missing UIs? To drop it and create a new application?

Would be good that you specify a bit how you see that work in relation with the AP roadmap.
Right now we already have in AP a small UI for messaging: do you think we’d drop it in favor of the messaging UI right away?
Also most of the AP interactions right now are performed through Notifications: so we’d need to be able to react on Notifications. Do you plan to have a way to start a discussion from a notification directly in the messaging UI as part of the “main messaging UI”. Else I do think the quick actions on notifications should occur quite fast so we have them for AP.

Thanks

#12

IMO, regardless of the backend, the scope of the messaging application is limited
and it is not critical to keep the existing code.
Though, it is interesting to keep the namespace to allow wikis to be upgrading to the new messaging application seamlessly.
But I think it is easier to drop its support and create a new discussion/messages module and provide a migration from message stream.

Indeed adding an entirely new store is probably not the best option. Regarding solr, this is a viable option to consider, especially if it offer some advantages regarding messages indexation.
But afaik, it would provides less useful features out of the box.

Yes, especially since IMO the integration is a very good use case to ensure that the discussion module implementation is generic and extensible.

This is not mandatory and we can for instance provide a way to start a discussion from a notification directly from the discussion app main UI.
But the need to quickly react and answer to notifications has been expressed by stakeholders on the context of the mentions design, so while it is not strictly mandatory for AP, I think it is interesting to take the opportunity to work of that feature.

If we all agree on the general direction of this brainstorming, I propose to start two design pages for the discussion app and the notification quick actions, detail further the UI and Architecture of these features.

#13

Hi everyone,
we brainstormed with Manuel yesterday during a call, and we are starting to wonder if this Discussion Application wouldn’t be actually a big improvment of the existing Forum Extension, the improvment being the ability to link a topic with an XWiki Event.
@acotiuga would you have any input on that since you know best the Forum Extension?
[Edit: Note that we started to discuss a bit about it with @lucaa too on XWiki Chat: https://matrix.to/#/!ikPtGZaGWtyblizzlR:matrix.xwiki.com/$16025859134200AIxHe:matrix.xwiki.com?via=matrix.org&via=matrix.xwiki.com&via=t2bot.io]

#14

I think the idea of using the forum as a store for discussions is interesting. However, they are not exactly the same features. I mention “store” because discussions should be in-place (i.e. contextual) and the forum topics are not contextual (you access them through a main forum UI listing all forums and all topics for each forum). So while the forum provides a UI, we would probably need different contextual UIs too. But these UIs could save the discussion data in forum wiki pages, using forum xobjects I guess. Then there’s the problem of mixing forum topics with contextual discussions. I guess we’d have a special forum dedicated to discussions and each thread would be mapped to a source (ideally an entity - wiki, space, page, xobjects, xproperty, etc -. However ATM a notification is not an entity so it could be more than just entities, could be a Resource as in xwiki-platform-resource). Maybe this dedicated forum would even be “hidden”, i.e not displayed in the list of forums so that it’s not confused with other proper forums?

#15

Side note: I don’t think this type of forum-based discussions should be bundled in XS.

#16

Not sure to follow you entirely there: IMO we do need a central place where to store and displays the discussion, that was even one of our main argument to have a dedicated Discussion Application, see:

I agree that it would be nice to have the display of discussions in context, but for me the most common usecases would be that you open your notifications, and on a notif you see that a discussion started around it. I don’t think it would be a good idea to display the entire discussion there.
Maybe you have other usecases in mind?

I agree with that one: IMO we can imagine reusing the forum APIs, but the UI should be entirely new for the Discussion app, so we wouldn’t have this problem.

WDYM exactly? That you don’t think a discussion app should be bundled in XS? Or that if we based it on the forum extension, and only in that case it shouldn’t be bundled?

#17

Yes, we agree. I was thinking about discussion about a wiki page for example. The best place would be in a Discussion tab at the bottom of that page.

Now, indeed, it would be nice to have a filtered view in your user profile to list all discussions you’ve participated too (with searching) and be able to navigate in place to see it in its context (ie with the page content in this example).

Maybe we’d need a new feature in the Forum app to be able to embed a context (summary or fully) so that you can follow the discussion without navigating away. Also, in some cases the context will have disappeared (imagine a notification that’s been removed for ex or a delete wiki page) so it could be good to have a copy of it in the forum topic (or a summary).

#18

I’ve check the forum and forum pro application and they provide what we need in term of user discussion features.
I think we can also discard the message stream application as an alternative since it is too minimalist for our needs.

In the context of what we are brainstorming (and summarizing what has been discussed aboce), starting a discussion could be creating a new topic with a description containing the description of the discussed event. In addition we should allow the topics to store a reference to the initial event (or maybe some other kind of entities too?).

If we go in this direction, we have to choose the forum in which the discussion related to the event must be added.
Brainstorming:

  • let the user decide when creating the discussion
  • define a default forum for the event discussions
  • define a hidden forum for the event discussions, displayed only in a separate, dedicated page

If we go in this direction, that also probably means to make the forum support ActivityPub.
That also mean to define a mechanism to display forum topics contextually (for instance, at the bottom of a page to replace the comments, or in ActivityPub to replace the current messaging system).

Pros:

  • The forum is a good starting point in term of functionalities
  • The question of the storage is already addressed

Cons:

  • The changes might be quite intrusive of the forum
  • Not mandatory but that might also mean to ship the forum in XS.
  • I’m not sure the forum currently support the kind of visibility we want

PS: Of course we can also consider to extract some features from the forum application to reuse them for the development of the discussions.

#19

Do you have any idea what would be the cost to allow “instantiate” Forum Apps with a dedicated UI? My idea above was to be able to reuse the Forum Data structure and possibly API (if there’s any), but without interfering with the existing Forum UI. So pretty much like the job I’m doing for Ratings: to be able to instantiate different “Forum Apps” with dedicated UIs: one Forum App for the regular Forum Extension, and another Forum App with a very specific UI for the Discussion Application.

#20

I did a quick analysis of the entities involved in the forum.

The forum application is composed of multiple forums, a forum has multiple topics.
A topic has multiple answer and an answer can be commented using XWiki comments mechanism.
The forums, topics and answers each have a single description field.
Only the topics have two additional fields: type and status, respectively using an enumeration of some predefined values.

I will do a second pass in the afternoon to see if we can reuse what’s existing to display and post answers to a topic outside of the forum application.