Organizing a large project

[Faille]

Originally Posted by Froofy-D

Haha the VG jokes never get old.

Actually, the content folder is the vast majority of the MB size of the game (music, textures, models). However, there are no lines of C# code in it. Content is basically just raw data.

that's what Sigil said!

[slitz]

Originally Posted by rangoth

Yes you can optimize once it's all said and done but you are just making it more difficult on yourself. Do it at the micro level as much as possible.

All others are good suggestion (but obviously SVN instead of CVS as said already), but this is a terrible one.
There is absolutely no benefit at all to optimize as you go since you're most likely ending up spending way too much time on code that wouldn't matter in the long run.
Unfortunately, it's damn hard to get past this since one actually think it's for the better. In fact, I'd say most university lectors / professors etc (that haven't been in a "real world project") would encourage you to optimize and give you a better grade for doing so, but in "real world programming" you would look incompetent.
30 lines of code is a LOT better than two, if 30 lines describes it better.
Unfortunately, you won't realize that fully until you get some hands on experience and get to take over someone else's code and / or get thrown into a already started project.

Unit testing is awesome however, but as said already it's a fairly complicated task since writing good test cases are far from trivial. But if you follow rangoths advice and accept that your coding will take five times longer in the beginning, the reward will be huge.

Good design and scalability are the two most important aspects of large projects (unless you're doing TDD but then I guess the good design comes with the territory, or hopefully anyway), but I guess they are also what makes large projects hard to cope with.
Unfortunately, you need a good theoretical understanding and lots of practical experience but I can at least give you a few pointers that I try to follow on a daily basis (bet these are all over the other forums though, from me and other developers so it's basically just a summary I guess):


-- Note, there are exceptions to all these cases, this is just in general.

(1) Program to the interface, not the implementation

- If you for instance need a parser for your game engine, make a parser interface first of all and let your implementation "implement" the interface. If you follow this rule,
it won't matter what kind of implementation you have your code won't care since it just follow the rules of the interface (and thus you avoid ugly typecasts).
The intention is to hide the implementation from the caller, aka loose coupling. The caller shouldn't have to know the inner workings of the function/method, the result is the only important thing (much like a black box if you will).

ParserInterface parser = new XmlParser();
Object ob = parser.callInterfaceMethod(); 
// Object ob = ((XmlParser)parser).specificImplementationMethod(); <--- ugly and none scalable solution

(2) Typecasting is normally a sign of bad design at interface level. See first point.

(3) One task per class

- If you end up with classes with 400+ lines, you're most likely skipping this golden rule. DON'T make a class that parse a file and then let it go through a formula to get the randomness of a epic drop for instance. That is not the OP way.
Make one class that handle the parsing and one class that handle the mathematics. Don't even include formating in the math/formula class, that should be yet another class task.
That way you can change your implementation (format, math or parse) at any time without having to change the rest of your code

(4) Don't overuse global state

- While singleton is a very powerful design pattern, it should only be used when it's absolutely necessary and when it should only exist one instance of the object. Whatever you do, don't turn it into some "global state" object. This is especially bad for testing.

(5) Separate the business logic from all your other program logic

- Guess (1) and (2) already covers the main points of this, but it's still worth noting since it's so darn important.

(6) Use the constructor as much as possible

- I see a lot of the following code:

private Object myVeryImportant object;

public MyClass(){}

public void setMyVeryImportantObject(Object myVeryImportantObject){
this.myVeryImportantObject = myVeryImportantObject
}

public void init(){
 // do something with the important object
}

This is absolutely awful. Yes, sometimes it does happen that your object can't get a reference to the important object on creation, but it's mostly due to design and not programming logics. The above should never ever be done unless you have a dependency injection mechanism at hand.
If you encounter the situation (and/or have a init() method), your design is most likely flawed. The initialization of the object should be taken care of the constructor (obviously enterprise frameworks are exception here).
If your object depend on other objects, you should force the initialization of those other objects, by adding a argument constructor and removing the default one.
Yes, this is something you learn in your first OOP class, but you lose it after 2-3 years of coding and then get it back again after 5-6 years 8). So avoid that cycle and just stick to it from the start till the end....
Your unit tests and future testers of your projects will love you for it 8)

(8) Learn and Love MVC

Model–view–controller - Wikipedia, the free encyclopedia
I would think this apply especially to games

If you follow these rules and do unit testing, you're a lot closer to a "successful" project than if you don't.

[rangoth]

Originally Posted by slitz

Unfortunately, it's damn hard to get past this since one actually think it's for the better. In fact, I'd say most university lectors / professors etc (that haven't been in a "real world project") would encourage you to optimize and give you a better grade for doing so, but in "real world programming" you would look incompetent.
30 lines of code is a LOT better than two, if 30 lines describes it better.
Unfortunately, you won't realize that fully until you get some hands on experience and get to take over someone else's code and / or get thrown into a already started project.

Allow me to clarify since people seem to keep misunderstanding what Im saying...or maybe its just a genuine difference of opinion.

First off, I have worked on huge, non-academic projects for a number of years now. I've been a lowly programmer on them, lead engineer, manager, designer, etc. as I have worked up the chain.

What I mean by "early" optimization is to do things in the best possible way from the start. I am not trying to say you should sit down with every function and try to turn 50 lines into 2 cryptic lines because you save .5secs per execution while making your code impossible to understand.

For example. If you are coming up with an image reading/manipulation aspect of your application. Think about how you are doing that. Optimize the loading of the image early, the manipulation of it early. Do not wait until your entire project is done to learn that you are editing the image in reverse order(think about how images are loaded into memory and read them in a more logical way). Or do not wait until your project is done to learn you can load the image in a seperate thread so not to waste resources. Those are the kind of things that when you try to shimmy them in at the end you now have thread locking and synchronization issues with. Now have fun trying to solve that in your huge complex application. That is, assuming you even have the proper tools to accurately determine where the exact time is being spent(you'd be shocked how many people don't have these kinds of analysis tools).

I guess I should change my wording from "optimization" to "intelligent coding" or something to make people more happy.

[slitz]

Originally Posted by rangoth

stuff

First, I have to note here that when I was talking about "real work experience" it wasn't directed at you, it was just in general. I don't know your background nor would I assume to know it 8)

Anyway...

It kinda depends on if I would call that premature optimization or not.
What kind of application are we talking about? Is image manipulation one of the key aspects of the product or just a tiny feature? If it's a tiny feature, I'd say that optimizing the way it's written to memory is premature optimization.
If the standard API offer a way to buffer it, but you yourself can (in your opinion) optimize it so it uses 20% less memory and load 20% faster, I would still call it premature optimization, because it's done under the assumption that it will effect the program overall performance (even if it's a key aspect of your application).
How can one possibly know this that early into development, even if it makes loading the image faster?
One might spend 10-15 hours optimizing something that turns out to be a piss in Mississippi when a profiler is used (for overall performance), when one could have just ignored it for the moment and let the profiler find it and then solve it. Instead, one wasted 10-15 hours on optimizing something that didn't need optimization, 10-15 hours that could have been spent on stuff that actually needed it.
Obviously, all numbers are just examples, but you get my point.
With that, I'm just trying to get my point through, then you'll have to reflect yourself if I disagree or agree with you.

Of course, if you're coding mostly from scratch (how often does this happen? 8) ), one should of course use the API to its best ability, but I wouldn't say that's about optimization either, it's about knowing the API.
Even so, one still shouldn't sacrifice readability / good design for optimization.

If you're an framework writer however, then all those arguments are completely void.

But threads/synchronization is imo about knowing how to program, it doesn't really have to do with optimization (ie making sure threads aren't waiting too long for locks to be removed)

[Punko]

A+ thread.

[Throag]

Originally Posted by slitz

Of course, if you're coding mostly from scratch (how often does this happen? 8) ), one should of course use the API to its best ability, but I wouldn't say that's about optimization either, it's about knowing the API.

It's not only knowing the API, it's also knowing what the hardware does behind the code. I'm saying that because I work on game consoles and performances are critical. When you absolutely know that the piece of code you are writing will be critical (it happens :P) you have to think ahead and make it as hardware friendly as you can (cache friendly memory usage, avoiding unnecessary copies, LHS...). This is the kind of problems a lot of people just totally ignore. Of course it really depends on what kind of application you are working on, but in these cases I would not call it premature optimization, but intelligent writing

Anyway, I think we pretty much all have the same global view of the problem

[slitz]

Actually I would like to elaborate on the whole "unit testing" thing rangoth brought up. Of course, no one asked about it but I recall myself that I never learned about it at the university and yet I had to run a couple of unit tests at my first day on my first job. I didn't have the slightest clue on how to do it so I ran to the local book store and bought "Planning Extreme Programming" and it totally changed my way of planning projects and implementing them.
I guess the mantra may have changed since then though, but I'll try to sum up something called TDD, Test Driven Development which comes in handy in your everyday programming and at work.

Basically, the whole idea of it is to do tests before you implement your code.
First do a test that fails, make sure it fails and then do a test that succeeds with as little code as possible.

As an example, I'll use a "customer database":

Requirement 1:

One is supposed to be able to save customers to a list and add new customers to the list. The order of the elements in the list is irrelevant.

What to do?:

Make a list that holds customers

Tests of the List:

Test 1: An empty list have the size of 0
Test 2: Add a customer to an empty list and the new size is 1
Test 3: Add two customers and the new size is 2
Test 4: Add a customer and then look if it contains the customer

Implementation:

Since I'm a java programmer I normally use JUnit for testing so all the code will be in Java and JUnit. It's not really hard to understand in either case.

public class TestCustomerList extends TestCase {

   public void testEmptyList(){
        CustomerList list = new CustomerList();
        assertEquals("size should be 0", 0, list.size());   
   }
 
   public static void main(String[] arguments){
        junit.textui.TestRunner.run(TestCustomerList.class);
   }
}

Running this code will result in error as intended since there isn't a class called CustomerList.

public class CustomerList{
    public int size(){
        return 0;
    }
}

Running the test again results in a successful test.
Do notice that the size method is completely useless at this stage, but that's also the point of this kind of programming. Only add as much code as needed to pass the test.

Next test is to add a customer:

   public void testOneCustomerList(){
        Customer c = new Customer();
        CustomerList list = new CustomerList();
        list.add(c);
        assertEquals("size should be 1", 1, list.size());   
   }

Assuming we made a class called Customer (that's basically a completely empty class at this stage), and add a empty method to the CustomerList called "add" (it should be a skeleton method) the test will fail since the size isn't increased.

Making sure the next test succeeds:

private int numberOfCustomers = 0;

public int size(){
   return numberOfCustomers;
}

public void add(Customer c){
      numberOfCustomers = 1;
}

The second test now succeeds after adding a customer.

Test 3:

   public void testTwoCustomerList(){
        Customer c1 = new Customer();
        Customer c2 = new Customer();
        CustomerList list = new CustomerList();
        list.add(c1);
        list.add(c2);
        assertEquals("size should be 2", 2, list.size());   
   }

Obviously the new test fails..

public void add(Customer c){
       numberOfCustumers++;
}

Third test passes and we jump to last test

public void lookForCustomerTest(){
      Customer c = new Customer();
      CustomerList list = new CustomerList();
      list.add(c);
      assertTrue("should contain new customer", list.contains(c));
}

Obviously the test fails since it's missing a contains method. Adding the method:

public boolean contains(Customer c){
        return true;
}

will make sure the test succeeds.

Now obviously, I'm missing a lot of test cases here for the list, but it's only a starter and it's also on a very simple level just to show what TDD is all about.
The real challenge about TDD is to actually come up with the test cases that make sure your classes work as intended, but often that task is in the hands of the testers or test managers.

But after 50k+ lines of code you end up with a huge bunch of tests and if you follow these simple steps (after 50k+ lines, you still run ALL THE TESTS, including the first most trivial one and thus you notice if a test fail on each iteration) you won't have a single bug in your program (according to the test specification at least).
Of course, this type of programming takes a long time, so the development time for the first release candidate might actually be increased by 300-400%, but as we all know, that is nothing compared to the time it takes to support a software full of bugs. And teams that use this kind of practice often end up using less time overall on a project due to the few number of bug fixes needed.

Anyway, the above is just scratching the surface of TDD obviously and it's a lot more to it than that, but thought it was worth bringing up in a discussion like this.

[rangoth]

Originally Posted by slitz

more stuff

I do agree with you there. The ever favorite resource triangle must come into play. There must be a business case for everything. So going over quotes time estimates for a small aspect of the program is obviously a waste. I just used the image example because it is nearly the entire aspect of one of the products we released.