docs/testing/manual-testing.md
As a general rule, we like to have automated tests for everything that can be practically tested. However, there are certain types of bugs that are best caught with old fashioned manual testing (also called manual QA). Manual testing not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics of a feature they're working on.
This doc assumes you know how to set up a local development server and open the Zulip app in the browser. It also assumes a basic knowledge of how to use Zulip.
When testing Zulip manually, here are things to focus on:
You generally want to test with Cordelia as the primary user, and use Hamlet as her primary conversation partner. Use Iago when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things such as Cordelia not being able to receive messages not intended for her.
The rest of this document groups tasks into basic areas of functionality of the system. If you have multiple people testing at once, you can divvy up QA tasks by these sections in the doc.
We mostly test the message view as part of testing everything else, but there are few things to specially test here.
Try using all the navigation hotkeys:
Try narrowing from the message view:
With messagebox we want to test mainly the functioning of all all the keyboard shortcuts and click handlers.
Apart from that there are three views of a message box, we want to test their appearance too:
Here's how we're going to test the message appearances:
/me test message)
For all the three cases, we need to test the click handlers and the hotkeys too:
Play with the screen size to check if the messagebox appears fine in different screens too.
With message editing we mainly want to exercise topic changes.
Here are some tasks:
Do lots of editing
Test UI entry points
Zulip uses the term "narrowing" to refer to opening different views of your messages, whether by clicking on sidebar options, recipient bars, or by using search. The main focus of these tasks should be watching unread counts. Of course, you also want to see messages show up in the message pane. And, finally, you should make sure that no messages outside the narrow show up in Cordelia's view.
:::{important} Make sure that Cordelia is subscribed to Verona but not subscribed to Denmark; if not, you should use different channels for your testing. :::
When testing narrows, you want to have Hamlet send the same message several times in a row, while cycling Cordelia through various narrows.
Here are the main tasks for Hamlet (and each message gets sent several times):
For each of the above types of messages, you will want to cycle through the following views for Cordelia (and have Hamlet send new messages after each narrow):
There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then the whole exercise is about 30 minutes, assuming no bugs.
We have pretty good automated tests for our Markdown processor, so manual testing is targeted more to other interactions. For composing a message, pay attention to details like what is automatically populated and where the focus is placed.
Hotkeys
Buttons
Topics
Formatting stuff
#**devel** syntax and send to Hamlet, then follow the link.Attachments
Drafts
Click to send
For this task you just want to go through all of our popover menus and exercise them. The main nuance here is that you occasionally want to click somewhere on the UI outside of an existing popover to see if the popover menu is "too sticky." Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view, then the message will disappear from the view.
Here are the things to test:
Channel sidebar menus (click ellipsis when hovering over channel filters)
Topic sidebar menus (click ellipsis when hovering over topics)
Left-message-pane menus (click on person's name)
Right-pane-pane menus (click on chevron when hovering)
Buddy list menus (click ellipsis when hovering over users)
This is a fairly quick task where we test the search filters on the left sidebar and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to that channel.
Channels filtering
Buddy list filtering
This is an important category to test, because we obviously do not want to have bugs where people can read messages on channels they should not have access to.
The general flow here is for Hamlet to create the channels and verify that Cordelia has the correct visibility to them.
First, we start off with "positive" tests.
For negative tests, we want to dig a little deeper to find back doors for Cordelia to access the channel. Here are some techniques to try:
For public channels, it's ok for Cordelia to know the channel exists, and she can subsequently subscribe. For private channels, she should not even know they exist (until she's invited, of course).
channel:foo.The main task for testing search is to play around with search suggestions (autocomplete). Once you select an option, verify the message view is consistent with the search and that the left sidebar reflects the current narrow. If a search comes up legitimately empty, have Hamlet send a message that matches the search.
Here are searches you should be able to do with autocomplete:
There are some things you can try that don't come up in autocomplete:
Miscellaneous:
Test various UI entry points into channel settings:
Create new public channel "public1" and add Hamlet:
Test subscribe/unsubscribe:
As Cordelia, exercise different options in Create Channel dialog by creating channels s1, s2, s3, etc.:
Test per-channel options:
You can modify per-user settings by choosing "Settings" in the gear menu. Do these tasks as Cordelia.
We mostly test keyboard shortcuts as part of other tasks.
Here are the tasks for this section:
Make sure that these options launch appropriate help screens:
Here are the tasks:
This document does not cover settings/admin options yet. The main things to do when testing the settings system are: