My bachelor’s degree was mostly about proving that I can build things. The master’s degree added the challenge to prove, that I can also approach those things scientifically. That’s why one part of my studies was an 18-week long research phase dedicated to a single scientific project. And unlike many of the projects I had worked on before, there was no team this time. Just me, a topic and two examiners accompanying the project.

So before writing a single line of code, I had to answer a rather important question:

What do I actually want to research?

For me, one thing was clear from the beginning. I did not want to spend 18 weeks on something that would simply be forgotten once this module was over. Ideally, whatever I built and learned during this time should become a foundation or at least be reuseable for my master’s thesis later on.

That made finding a topic a bit harder. My bachelor’s thesis had been heavily focused on tool development, but finding a project in this area with enough room for an actual research question could be complicated. So instead of starting with a specific tool idea, I had a look at the research fields of our professors from whom we needed to choose one as our primary examiner.

And that is where Virtual Reality entered the picture. I had already worked with VR during my internship and in personal projects, so the technology was not new to me. More importantly, one of my professors was researching in this area while also being interested in the Unity Engine, Software Development and building tools that improve existing workflows.

That sounded suspiciously close to the kind of project I was looking for. So we sat down and talked. Some previous research fellows of my professor, currently working researching Life Sciences at the University of Konstanz already had quite an interesting technical foundation available. On one side was VANTED, an established desktop application my professor helped building, for creating, editing and analysing biological networks. On the other side was GAV-VR, a Unity-based framework designed to visualize and explore networks inside Virtual Reality.

Which led to a question that sounded almost too simple:

Why not just take the networks created in VANTED and continue analysing them in VR?

Existing research already showed that immersive 3D environments can provide interesting advantages for the exploration of complexe networks. But it also suggested that some tasks remain more efficient, intuitive and generally more suitable for conventional 2D environments.

So instead of replacing an established desktop workflow with VR, the idea was to combine both worlds. VANTED would remain the place where networks are created and mostly edited. GAV-VR would extend that workflow into the third dimension and incorporate immersive analytical functions. Easy enough right? Well…

Why not just import the network?

At first, that was pretty much what I thought. VANTED can export networks. GAV-VR can import networks. Put both together and the project is done. Except that this only works as long as nobody actually tries to use it.

A manually exported network is nothing more than a snapshot. Change a node in VANTED? Export it again. Add something? Export again. Move something in VR and want that change back on the desktop? Well now things get interesting. And that was the point where the project stopped being about displaying a network in VR and startet becoming something more interesting like “how do you turn two completely separate applications into one connected workflow”?

While previous research had already proposed such hybrid workflows conceptually, my task was to find out whether it could be actually built and to evaluate if it could be used as the prior research intended.

What is GAV-VR Link?

The result of my 18-week project including, research, development, planning and executing user tests and documenting my findings, was a system called GAV-VR Link. It is not a single software but rather two separate add-ons, one for VANTED and one for GAV-VR, each implemented using their respective add-on architectures.

At its core, the system connects both applications and synchronizes biological networks between them. The development was split in two main phases each resulting in their own finished prototype. Phase one was rather primitive. VANTED already supports exporting networks as GraphML files and GAV-VR already has a way of importing such files. So technically, I could export a network, switch applications, import it into GAV-VR and display it there. Behind the scenes there was a lot more work required for GAV-VR to interpret each VANTED attribute correctly.

But overall it worked. But it was also terrible to use. As I described earlier, every change to the network would require another export and import.

So for phase two, or the second iteration, I added a local webserver to the VANTED add-on and a corresponding client to GAV-VR. Instead of relying on manual file transfers, networks could now be sent automatically from VANTED to GAV-VR. Once the inital network has been transferred, individual content or attribute changes are then synchronized between both applications.

And most importantly, it works both ways. Moving a node in VR can update its position inside VANTED, while changing attributes inside VANTED can immediatly update their representation inside GAV-VR.

To make this possible, I built a modular service architecture around the server communication. Different services handle different things such as attributes, added or removed network elements or other network changes. This kept the integration into the existing VANTED codebase relatively small while making it much easier to extend the system later on.

Of course, transferring data in real time, comes with its own problems. Dragging a node around for example produces a large amount of position updates. Sending every single one immediately caused changes to pile up faster than th receiving application could process them. The solution was a simple batching system. Changes are collected over a short interval and only the most recent state is transferred. So instead of desperatly sending hundreds of outdated positions, both applications only care about where the node actually ended up at that point in time.

From 2D to 3D

Transferring the data was only half the problem. VANTED is a two-dimensional application. There is simply no third corrdinate waiting somewhere inside the network that can magically be used once the data reaches the three-dimensional VR space.

The system therefor had to preserve the original 2D layout while creating some useful representation in three-dimensional space, based on the content and the semantic of the network. For example what 3D form does a circular 2D node result in. Is it a Sphere? Is it a Cylinder? There is no definitive solution to this question since the 2D data simply lacks all information required to make this decision. Decisions like those should therefor be made context- or rule-based.

Once a 3D-reprensentation was found, additional information like the node’s Z-position or 3D-form, can then be stored alongside the original network data, allowing changes made in VR to persist instead of disappearing the next time the network is loaded.

At this point the basic system worked. But a simple working technical framework alone does not really prove that this system can be used to add actual value to data-analytic tasks. For this i required actual use cases.

SBGN Clones

SBGN, short for Systems Biology Graphical Notation, is a standardized visual language used to represent biological networks. It defines how different biological entities and relationships are displayed so that a network can be interpreted consistently across different tools and applications. One concept within SBGN are so called clones. In larger biological networks, the same entity can appear in multiple places. Instead of drawing long connections across the entire network, the entity can be represented by several nodes that are marked as belonging to the same underlying element.

This helps reduce edge crossings and keeps complex networks more readable, but it also means that a single biological entity may be scattered across multiple places in the network. In VR, however we suddenly have this completely new, up to this point unused additional axis.

So I experimented with using the third dimension to rethink how these clones could be represented. Different visualization modes move clone nodes onto dedicated depth layers, combine multiple clone representations into a single node or calculate their position based on the spatial distribution of their original counterpart.

The goal here was not to find THE one correct visualization. Instead, I wanted to explore how the additional spatial freedom of VR could offer completely new ways of handling problems differently compared to the two-dimensional space.

 

Stacking Networks

The second use case deals with comparing multiple networks. Instead of placing several networks next to each other on a flat screen, they can be arranged as a 2.5D network stack. Each network becomes its own layer, while the entire stack can still be moved, rotated and inspected as one object. Individual layers can also be repositioned to help compare structures between different networks.

Both use cases were built to see whether the architecture could actually serve as a foundation for new analysis tools instead of becoming a highly specialized connection that only solves one particular problem.

Scientific Value

To finish the project, I evaluated the prototype with three experts in biological network analysis. The evaluation combined questionnaires with a Think-Aloud approach while participants worked through a series of tasks in VANTED and GAV-VR. Given the very small number of participants, this was never meant to produce statistically meaningful results. Instead I wanted to find out whether the system fundamentally worked and identify where future work should improve upon this concept.

The actual transfer from 2D to 3D worked fine. Participants considered the spatial representation sufficient similar to the original networks and both experimental use cases showed potential in making the analysis process more efficient and intuitive. The biggest problem was not really the visualization but rather the transition between desktop and VR. Some functions would be only accessible in 2D while others where exclusive to VR resulting in having to take off the headset, interact with VANTED and then putting it back on. This interrupted the workflow significantly more than I initially expected. Some interactions inside VR also still lacked the usability and polish required for productive everyday use, based on this being a research prototype.

Instead of simply concluding that “VR makes networks better”, the project showed that the way desktop and immersive environments are connected may be just as important as the visualization itself. This opens up an entirely new set of questions around hybrid workflows and Transitional Interfaces and gives the project quite a log of room for future work.

Project Report

As part of the project I was tasked with writing a complete technical report documenting the motivation, related research, system architecture, implementation, use cases and evaluation in much greater detail than I could reasonably fit in this post. The report was originally submitted as part of my research module at University of Applied Sciences Mittweida and has not and will not undergo a formal peer review.

I have now made it publicly available on GitHub under CC-BY-4.0. If you are interested in the technical details behind GAV-VR Link, the architecture connecting VANTED and GAV-VR or the results of the expert interviews, feel free to take a look.

Read the Technical Project Report →

And how do I reuse these findings and the created system?

The project will not end with this module. In a few weeks, I will start working on my master’s thesis and GAV-VR Link will serve as its technical foundation. While the thesis itself will take the project in a quite different direction, I definitely want to return to the emerged questions about Transitional Interfaces in future research.

With this the project became exactly what I hoped it would be. It is not an isolated piece of coursework, that will never be used again, but a starting point for my next big step towards my final degree.