Tuesday 29 March 2011

"Is it too early for commitment" said the developer


"Stories are part of every community. They communicate culture, organize and transmit information. Most importantly, they spark the imagination as you explore new ideas. They can ignite action."

-Whitney Quesenbery


I am looking for all KDE developers who have had to deal with program data requirements to give me their opinion on an issue I have and hopefully we can come up with an inovative way to solve the problem, in true KDE style. If you do not have time to read the full post please skip down to the technical question and add a quick comment. 


Let me tell you a story. A college student studying Computer Science decides one day to download the source of his favourite Open Source program and build it from scratch. This college student wants to impress his friends by adding a simple button that shows a dialogue box with his name in it, hence proving his l33t skills.

This "user story" sets up a persona for the protagonist of this blog post, a very low commitment developer that is trying something on a whim. The user story also paints an interesting picuture of the requirements of the user:
  1. Download the program quickly and easily.
  2. Make a really small change, quickly and easily.
  3. Run the program quickly and easily.
I'm adding the "quickly and easily" because I assume that college students have many other valuable things to do with their time... and time is very valuable during college.

So taking a well known KDE program as our test subject, how does this scenario fit with the Marble Desktop Globe code base?

1) Download quickly and easily:

The Marble website has a link on the frontpage to "Compiling Marble". Well placed and on a very nice looking website! My college student is having a good time so far, he downloads the source from the Subversion Repository and thinks to himself "Hey I like IDEs so why don't I just skip the next part of the instuctions and run marble from, say, QtCreator" ... move on to step 2

2) Make a really small change, quickly and easily:

This is a topic for another blog post. I believe that code be structured in such a sensible way that it will pratically guide a new developer to where they need to make their change. More on this some other time ... move to step 3


3) Run the program quickly and easily:

My student is loving his development experience so far. He's enjoyed looking at screen shots on the Marble website while downloading the source, he's found the part of the code he needs to change really quickly and has made his simple change. So what is there left? Just click that big play button in QtCreator and show off your skills to your friends. Wait a few minutes for it to compile and hey presto! I give you marble:





Eh.. Ok What happened here? Our student is confused and a little frustrated but he make at least a tiny bit of effort to see if he can find out the cause of the problem. He finds this message:



At this point the ease of development has completely disappeared for our imaginary student. What he doesn't know is that Marble is expecting its supporting data to be located in a specific place, and if he read the rest of the compilation instructions he would have seen that you have to run make && make install before running marble.

Now our student has a choice:
  1. Continue diving into the code of something that he doesn't understand to figure out the problem and fix it
  2. Get involved in the community by contacting someone on IRC or a mailing list (which he has to find) and wait for some help
  3. Give up.

Which do you think he will choose at this point? Which one of these outcomes do you think the *majority* of the population of mildly interested, not very commited people would do at this stage? What would you do?

I have brought this discussion up on the #kde-devel chanel trying to figure out a nice technical solution to the underlying problem in marble and I got the response from one of our KDE developers:

<*unnamed*>: i'm always concerned if we need those really non commited devels

Now the developer in question may make a very valid point. Why should we spend loads of time and effort on the people who think its too much effort to read the the full "Compiling Marble" page. It is a valid point but I fundamentally disagree.

I believe that our imaginary student should go away from this experience thinking that Marble is an easy project to develop for, and sometime in the future he will remember this fact. It might be days, weeks, months or even years but that developer will come to a time in his life where he has an extra 10 mins a day to work on some open source program and will remember his good experience with Marble. This has a powerful effect in the long run.


The thing that sparked this philosophical rabbit hole for me was a simple technical question: how do you make it so that a developer can just click "run" in QtCreator when working on a CMake based project where the program has data requirements? The current way Marble does it is that it requires the user to run make && make install before running, which to me is not optimal for 3 reasons.

  1. The uncommitted developer (our student from the user story above) may not read the documentation enough to realise that they need to make install before running.
  2. The uncommitted developer may not want to scatter files, programs and libraries around their computer without knowing that they will be able to clean their system afterwards.
  3. The default configuration of make install is that it places the install files in system paths that will most likely be security protected so in fact the user will have to run sudo make install.

Point 2 above is problematic for our uncommited developer because they, most likely, are not committed enough to read the CMake files and understand where the data will go when they do a make install. Point 3 can be avoided by configuring the install path but this is another piece of configuration our student needs to figure out or read somewhere, and frankly I don't even know how to do this.

So what do we do to solve this problem? Marble has a unique way of dealing with its data dependencies, it uses a class called MarbleDirs that is effectively a layer of indirection for the file system. It currently checks 2 places for any of the file names/paths that are requested

  1. The system install path for the data
  2. The user's application data folder, so that files can be created without administrator privileges

This system also serves the purpose of locating the "system install path" and the "user application data folder" in whatever OS Marble is installed on.

I have proposed a change to this system http://reviewboard.kde.org/r/6574/ that adds another folder to the "watched folder list". The new folder will be the application directory where the Marble executable is located. This ensures that if the data required by the Marble program is copied to the build directory at configure time (with CMake) the QtCreator IDE will run marble fine right out of the box.

I would ask anyone who writes any program that deals with a data dependency to please let me know what your program is and if you have the time a brief description of how you deal with this issue. If you don't have time just tell me where to find the code and I'll go have a look myself.

I am trying to get a feel for how other people have solved this same problem before I re-invent the wheel. Also if you have any ideas please chip in with a quick comment.

11 comments:

tbscope said...

Ohh dear...

The clasical lazy student.

It's like saying: "I want to make a quick and easy modification to the space shuttle. Whatever did I know I had to use a difficult to understand fueling method to get the d*mn thing into space?"

This is just a classic example of not wanting to read the documentation. There's no excuse for it.

You can build any program in any IDE. You just need it to be set up correctly, and that takes time and knowledge. You know, things that a student should be doing in the first place: learning.

A lazy student is just a lazy person, and a person who will fail in doing these tasks in later life (unless he changes).

Before you drive a car, lets first try to at least know how to walk.

That does not mean I say you can't start driving right away... but at least expect some bumps down the road and some detours.

I'm a professional electro-mechanical engineer. I have a lot of theoretical and practical knowledge. But guess what: every day, I need to learn something new, something I don't understand and something that takes time to know.

Deal with it. Either follow the given recipe, or actually learn something.

But: some things can indeed be made easier to understand, that is true.

Anonymous said...

Yes, this is a frequent problem. Not so much of the lazy developer, but oftentimes you have data that can and will be hand-edited in the source dir, and it should not be necessary to install in order to use it.

Basically, I have found 2 practical ways of dealing with this: 1. in Debug/non-release/some other suitable mode, include the path to the source dir and/or build dir 2. allow loading relatively to the executable. 3. make an environment variable and use that as a possible data dir.

1. has the advantage of not polluting the final application 2. has the advantage of working well on windows
3. has the advantage of being useful for some advanced users besides developers.

Note that both 1 and 3 requires the user to set up something.

P.S: You might want to suggest KDevelop as an alternative to Qt Creator.

Anonymous said...

I agree we should make it as easy as possible to start contributing on a new application. I believe if someone get an immediate reward first he is more likely to invest more time later.

However, I dislike having to support running applications uninstalled: it creates additional code paths, meaning more possibilities for things to go wrong.

Nowadays, "make install" on CMake-based projects is quite fast, so I would suggest the following:

1. Setup a desktop-wide, user-writable, additional install prefix for data.

Something like this:

export DEV_INSTALL_PREFIX=$HOME/opt/development
export LD_LIBRARY_PATH=$DEV_INSTALL_PREFIX/lib
export PATH=$DEV_INSTALL_PREFIX/bin:$PATH

# And for KDE development:
export KDEDIRS=$DEV_INSTALL_PREFIX:/usr

These variables should be set at login time.

2. Make it so that by default IDEs invoke cmake with -DCMAKE_INSTALL_PREFIX=$DEV_INSTALL_PREFIX

3. Make it so that by default the build target in IDEs is "make install" rather than "make" (maybe make it conditional to having a user-writable install prefix?).

With this setup, clicking "build" in your IDE should do the right thing.

Karthikeyan said...

Other issue with compiling programs is "dependencies". I myself have tried few times (multiple hours) and gave-up finally as some part of the compile would fail stating some Symbol not found, lib not found etc.

A good idea would be to have a meta package in the OS repo (ex: openSUSE or ubuntu) like "marble-dev-dependencies" which will install all the dependencies of marble. Not sure I am making any sense, just thought of sharing.

real_ate said...

@esben Thank you for your different suggestions. That is exactly what I was looking for from this blog post! Have you used any of your methods 1, 2 or 3 in an open source program or in a KDE program? If so can you point me in the direction of it, I want to check out some solutions in the wild. Thanks!

@agateau I actually quite like your technical solution but it still requires for the user to know to do something else before clicking play in their IDE. Would it be a possibility to *force* developers to set up those parameters? i.e. if the parameter DEV_INSTALL_PREFIX cmake will fail saying something like "Please set the parameter DEV_INSTALL_PREFIX". This way the cmake error log will suggest what to do to fix the situation.

@Karthikeyan your point made perfect sense and is very helpful. How about if CMake could not find one of the dependencies it could prompt something like "If you are running Ubuntu run 'sudo apt-get install marble-dependencies'". As I mentioned to @agateau above it might be good to identify issues and how to solve them in the CMake build output.

Anonymous said...

@Karthikeyan What I often do on Ubuntu before hacking on a program is to run "apt-get build-dep $program". This installs all the build dependencies. Of course it requires the program to be installed and it may fail if I am working on a new version which has new dependencies, but most of the time it works quite well.

@real_ate: I was thinking of something like a one-time setup. The distribution could provide a special package which would set up the development environment. This package could then be made a dependency of IDE packages. Since setting up the environment would need to alter user data (creating files in $HOME), maybe it could be included as a hook inside the IDE. Imagine the first time you start your IDE, you get asked if you want to setup a development environment, when this is done, the IDE can call cmake with the correct development environment options.

real_ate said...

@agateau I think you have really hit on something here. If you go back through my User story in the blog and in the part where they look at the Marble website it just says "Install this package and it will setup your entire development environment" that would probably springboard the new developer into contributing. Thanks for the idea, it will be going in my notes ;)

m35 said...

I've dreamed that someday someone would make a Linux "development distro": A virtual machine that would store your build environment. It could be configured to checkout the latest sources, manage the dependencies, and even cross-compile for a wide variety of targets. With just a single download, a new developer would have everything he needs to contribute. It would also protect the system of a casual contributor, either when development code goes awry, or just from being loaded with lots of "make install"s that he won't need next month.

Anonymous said...

No, I am afraid I have never used the technique in a (so far) published program. You are welcome to contact me (kde@mosehansen.dk) though, if you have any questions. All the techniques are pretty straightforward.

@it-is-so-easy-to-install: It is not the installing that is the problem, it is the fact that you have 2 different versions of the same data on your hard disk. You can and will edit the wrong version at some time, leading to unexpected behaviour.

glad said...

I used esben's points 2 and 3 to load the translations in the ktikz program. Of course if you need to load other things too, you will have to use a more generic approach than in ktikz (see app/main.cpp in the ktikz source). To obtain the current working directory of a Qt app, use QCoreApplication::applicationDirPath(). To obtain the value of an environment variable, use qgetenv(). Point 1 of esben can be done with the help of the cmake variables ${CMAKE_CURRENT_SOURCE_DIR} and ${CMAKE_CURRENT_BINARY_DIR} (I didn't test those).

PatsComputerServices said...

I like the idea of the "Debug" and "Release" options in the IDE, and also the idea of where the IDE configures everything for you (the build environment). In fact, IIRC, Visual Studio and Netbeans do these both of things for you (I know Visual Studio does for sure).

I also think that there should be a note on the website (for marble in this specific instance) recommending that the developer install the "marble-build-dependencies" or appropriate item--along with instructions on how to do this per their Operating System.

And speaking as someone who falls into the category that you described (to an extent), I also realize that if you don't already know how to install dependencies through your operating system (or have at least a basic idea of how programming works), you probably shouldn't just hop into this. Not because you're lazy or anything like that, but because it will be more than you anticipated. And you're more likely to just give up in frustration.

Have a great day:)
Patrick.