Creating a 3D Game Engine (Part 25)

So it comes with great reluctance but I think I will have to suspend development on my 3D game engine, at least for the near future. Surely this will be a disappointment to anyone following the progress and I did not make the decision lightly. After spending some time thinking about it, I think it’s the right move. It’s just really hard to justify the amount of time and effort put into this when there are much better solutions available off-the-shelf.

In the little less than 2 years I’ve worked on this project, I think I have made decent strides. Honestly, I was not working on it full-time. Just on weekends mostly or maybe here and there when I had time after work. My initial goal was to have something functional in about 6 months to a year. It took a little longer, but I guess eventually I did have the basic features of an engine somewhat working. I will list what was implemented below, but it was enough to load in some textured models and fly a camera around. Not ground-breaking work by any means, but I built nearly everything myself so I was happy about that. There was just an enormous amount of work remaining to get to a state where a game could be produced. Finally, it was too much.

So before I give my personal advice, I’ll break down how I got to this point. First, let’s look at all the books I read as research for this project:

Game Engine Architecture – Jason Gregory
Game Coding Complete, Fourth Edition – Mike McShaffry
Real-Time Rendering, Third Edition – Tomas Akenine-Moller
Game Engine Design And Implementation – Alan Thorn
3D Game Engine Programming – Stefan Zerbst
Game Engine Gems, Volume One – Eric Lengyel
3D Game Engine Architecture – David H. Eberly
3D Game Engine Design –  David H. Eberly
3D Math Primer for Graphics and Game Development – Fletcher Dunn
Mathematics for 3D Game Programming and Computer Graphics, Third Edition – Eric Lengyel
Introduction to 3D Game Programming with DirectX 11 – Frank Luna
Practical Rendering and Computation with Direct3D 11 – Jason Zink
Beginning DirectX 11 Game Programming – Allen Sherrod
Game Physics Engine Development – Ian Millington
Physics for Game Developers – David M Bourg
Physics for Game Programmers – Grant Palmer
Game Physics Pearls – Gino van den Bergen
The C++ Programming Language, 4th Edition – Bjarne Stroustrup
C++ Primer (5th Edition) –  Stanley B. Lippman
C++ Primer Plus (6th Edition) – Stephen Prata
More Effective C++ – Scott Meyers

If you are considering engine development, please read on, but I believe all the above books are valuable to look at. I’ve also read a variety of sources online, from tutorials to official documentation and more. Looking at that list it seems huge, but I am an avid reader and have read and listened to a number of non-technical books in that time-span as well.

Next, let’s look at what features were actually implemented and working (or mostly working):

  • Win32 app template (message pump)
  • DirectX 11 based rendering engine
  • DirectInput mouse and keyboard controls
  • Custom math library (Vector, Point, Matrix, view/projection)
  • Node-based hierarchical scene graph
  • Frustum culling (sphere and point tests, minimal bounding sphere)
  • Generic XML parser (from scratch, kind of hacky and slow)
  • COLLADA importer (only basic vertex and UVs)
  • Custom binary 3D model format (for speed)
  • Camera object (spectator mode)
  • Texture mapping and Skybox (cube mapping)
  • Directional and ambient lighting
  • Normal and shadow mapping shaders
  • Command-line console window logging
  • Architectural glue and probably some odds-and-ends.

So what wasn’t finished? Way too much to list. I don’t want to get down on myself too hard, cause I think that I accomplished a little something here. Certainly some of this stuff was not easy and, if anything, it has made me a better coder. Even in the current state I guess you could make a Pong game or a simple walk-through demo or the like. Also keep in mind this was a part-time excursion. If I were dedicated to this fully it probably would only have taken months (not counting research time). While I started with a very clean style and deliberate design, by the end I was kind of hacking things together, and some features (like the shadow mapping) became very intertwined with the core rendering code. Of course, this was very much a learning experience, and I planned to go back and clean things up later (which may or may not ever happen, we’ll see).

OK, so by now you are probably getting the idea. As to my advice, if you are trying to make a game (or game-like interactive experience) don’t bother making a custom engine. Everyone basically told me this from the start, and I didn’t listen. It’s sort of like when you see people doing stunts on TV and they tell you “don’t do this at home.” But they did it, and it made them rich and famous. So maybe I was looking for glory, or hoping finishing a big project like this would take me to another level. I guess in a way it did. I’m just not sure that was the best use of my limited time. What I mean to say is that knowledge is valuable, and I feel I learned a lot. However, unless you are really looking to be an engine programming professional, there is not much reason to try to do this.

The crux of the matter is that all-in-one engines like Unity and Unreal are just too complex and mature to expect to compete with. That’s not to say a small team or a single hero-coder can’t produce something great. They can. However, using a pre-built engine cuts out a *lot* of time and risk. Had I been working on a project for a client, I would have never even contemplated a custom engine. It just doesn’t make sense. Both Unreal and Unity have huge teams of expert engineers working solely on engine development. Unless you work for a huge AAA studio with a monster budget, I just can’t see coming up with something better in any reasonable amount of time. Even some huge companies (like Capcom and Square-Enix) are dropping in-house engine development for Unreal 4. Not a good sign for indies or bedroom coders working on their own engines.

But let’s say those big engines aren’t for you. There are a number of open-source engines and libraries out there to help you along. Something like Torque or C4 look promising (if you want a full-featured solution), or OGRE (if you don’t mind having to cobble together a bunch of different libraries). If you are interested in web publishing there are things like three.js for WebGL. With source access you can add the features you need or fix critical bugs. Really there are too many options to ignore.

For the last few weeks I’ve been playing with Unreal Engine 4, and it basically does everything I need and a whole lot more. At $19/month and a small royalty (if you sell your title) making a AAA looking game has never been more affordable. Unity may still have a leg up on ease-of-use and scalability on different platforms, but for high-end development I think Unreal is probably the one to go with. Having the source-code is a huge boon, and the price makes it attainable to almost anyone. Plus, with the visual scripting, Unreal can even be approachable to artists and designers. I’m still in the early stages of getting acquainted with the engine, but so far I am pleasantly surprised. I plan to follow up with some posts and tutorials in UE4 in the future, so don’t think this is the end of my blogging career or anything of that sort.

With all that said, there are still some things I’d like to see in engines that don’t exist today. You typically can’t use the editor in stereo 3D (assuming the engine even supports 3D natively), motion control support for placing objects is non-existent, image and model editing is usually very rudimentary (you must switch back and forth to other apps), real-time code editing and previewing can be limited. So things can certainly improve, and I guess there could be a case made for making plugins or forks to existing projects. I’m not saying things can’t improve. Just that reinventing the wheel to make incremental improvements may not be the best path.

At this point I don’t think I would say engine development is a “waste of time.” As a learning experience, I think it was fun and rewarding. Certainly I hope I didn’t just waste 2 years of my life. Doing all the work and research has brought me here to this place. So it was somewhat worthwhile. It just pains me to think how far along I might have been had I jumped on Unreal 4 when they first announced the subscription model, or spent more time with Unity over the years. If you actually want to build (and finish) a game, it’s a pretty clear choice to use an off-the-shelf solution. I’m sorry, guys. I wanted to prove everyone wrong, but they were right. Make games, not engines.

Finally, I’m sure the question you’re thinking is “will I release the source.” I’m considering it, but there are some hacked bits in there and I’m not sure it’s a great example to work off of. More likely I will pull out portions or classes from the project and use them for future blog posts highlighting a particular algorithm or technique. The full source itself is probably less useful compared to other stuff that’s out there. In any case, I hope maybe you have learned something after 2 years and 25 posts on this project. If you click the (somewhat hidden) 3 white line icon on the top right of the blog, you can see a list of all the blog updates in this series for a little history (if you weren’t following the whole time). Cheers, and happy coding.

Review: Game Programming Patterns by Robert Nystrom

Game Patterns

I will start by saying this book is game programming GOLD! Whether you are a pro or a novice looking to learn, this book deserves to place on your shelf (or I guess in memory if you buy the e-book). While some of the chapters may seem like obvious things for people that have programmed games before, I think even advanced coders will discover a few things they didn’t know.

So let me talk about what this book is. Basically it covers common challenges in game programming and some useful ways of resolving the problem. Though the theme of the book is game development, a lot of this stuff is applicable to any sort of visual or object-oriented programming. Nystrom starts by revisiting the classic design patterns popularized by the seminal book by the “gang of four” in 1994. Surprisingly, 20 years later a lot of those ideas still hold up. Next he moves onto more game specific topics like double buffering (not just for graphics), a game loop, and updating objects. Then he goes into bytecode (really a simple compiler), components, event queues, singletons, object pools, dirty flag and spatial partitioning. It’s actually not the longest book out there at 354 pages, but this is a breathe of fresh air after persevering through The C++ Programming Language (which was great, just very long). The author does not waste pages, though. There are nuggets of knowledge littered throughout the text.

One thing I like is how the book is not tied to a particular API or library. The pseudo-code is in C++, but really you could implement the ideas in almost any language. He even goes as far as not using the STL (for example, rolling his own linked list for a few examples). In a real application, you would probably not want to reinvent the wheel for basic containers, but it’s nice that the examples stand alone without any nasty dependencies. I could see a lot of the code here being copied into a real game and being usable with only minor additions. Well, of course you have to modify for your platform or engine or whatever, but the concepts are solid.

Another point is that this makes design patterns concrete (please, no abstract class jokes…). I read the original Design Patterns book years ago but some of the patterns never made sense to me. They were too abstract and, though interesting, sometimes didn’t click for me. This book, on the other hand, clicked the whole way through. Everything made sense, and was immediately clear why it was useful. Sure, I’ve probably learned a lot in the past few years, making Game Programming Patterns more approachable. But I think almost any game coder (or aspiring coder) could get value from this book. I’d give it 5 stars, 10 out of 10, 2 thumbs up, and definite “buy it now.”

Review: The C++ Programming Language by Bjarne Stroustrup

C ++ Programming

This book could be called “Everything You Ever Wanted to Ask About C++ and a Whole Bunch of Other Stuff You Had No Clue You Didn’t Know.” It really is the most comprehensive book I’ve seen on C++ and covers just about everything you will need to know (and maybe some more). That’s not surprising, as the book is some 1,300+ pages long. And, of course, it’s written by the creator of C++, so I would guess he knows a thing or two about the language.

The C++ Programming Language by Bjarne Stroustrup does cover all the major aspects of C++, but to name a few of the topics included: pointers and arrays, namespaces, classes, lambdas, overloading, copy and moving, templates (lots of templates), the STL, strings, regular expressions, concurrency, and the C standard library. It almost does a disservice to try to list all the topics, as it touches on almost all areas of the C++ language.  I found the code snippets to be useful and relevant, while still being short enough to be clear and understandable.

Keep in mind that this text has been updated to C++11 and the author does not bother teaching older standards. There are only a couple of instances when he mentions the difference but there is a chapter all the way at the end talking about some specific features that are different in each version of the language. This can be a good or bad thing depending on how you look at it. I mean, I think it was a good choice as the book would be twice as long and twice as confusing with constant clarification. But sometimes I was left wondering if a feature was new or if I just had never heard of it.

I’ll admit, I have actually not read earlier versions of this book, but I feel that Stroustrup has done a great job here. I would NOT recommend the book for beginners as I feel it is very technical and you might become afraid of C++ before even finishing the book. While the basic stuff about types and classes and all that are covered, the majority of the book is focused more on advanced features (templates, etc.). For intermediate to advanced programmers, this will be a great addition to the collection, and required reading to be an expert in C++. For novices I would recommend picking up C++ Primer or C++ Primer Plus (no relation).

One thing to note, this book is long. I think it may be the longest book I’ve ever read. Though I enjoyed it, there were points were I was doubting I would ever finish it. I would estimate it took around 2 months of daily reading to knock it out. If you wondered what I was doing since I haven’t updated the blog in a while, I was reading this. So before you embark on this adventure, make sure you’re fully prepared for what you are getting into. Recommended.

Creating a 3D Game Engine (Part 24)


 

It’s been some time since the last 3D engine update, but I’m still sticking with it. Currently I am working on getting a physics engine implemented. The video you see above is the first glimpse of this custom physics engine. Obviously it’s ultra basic right now, but it’s a start. The algorithm is based on a verlet integrator, and the code is running using DirectCompute on the GPU.

To be honest, it’s pretty hacked together right now, and the bounds/bouncing behavior is hard-coded. But, hey, it’s something! I also tweaked the style of the demo to more closely align with other demos from researchers (and the clean flat-shaded style looks a bit more classy).

I’m planning on expanding this significantly, but still have a lot of work ahead of me. Though I’m still questioning whether GPGPU is the way to go with this, especially since there are 8-core CPUs available for consumers (and which I am running now). There is also added difficulty and limitations of DirectCompute rather than straight C++. But if there are big gains on the GPU obviously I would like to go that route. Definitely more testing to do here, but progress is being made. Stay tuned.

Review: Game Physics Pearls by Gino van den Bergen & Dirk Gregorius

GamePhysicsPearls

Game Physics Pearls has been a book on my wishlist for a while, and I’ve finally got the chance to finish reading it and putting up this review. The text is edited by Gino van den Bergen and Dirk Gregorius, and each chapter is written by a unique author. I found that a lot of ground was covered while still keeping the book somewhat cohesive. It doesn’t feel like a complete random mash-up, and the progression is nice.

Some of the chapter content includes: basic mathematics, game physics pitfalls, broad phase, narrow phase, GJK, SPH, parallel particle simulation, ropes, soft bodies, and verlet integration. Even after reading several game physics books, there was still a decent amount of information I had not seen before. I especially appreciated the chapter on verlet integration (and the subsequent chapter on cloth physics that builds upon it). So many of the books I’ve seen seem to focus on Euler integration techniques and it’s rare to see much talk of verlet integration (or position based dynamics for that matter). This is the direction I am going with my physics engine, so it was nice to see some coverage.

Overall I was impressed with the quality of the book. Sometimes with these “gem” style books, it can be a hit or miss if the chapter is relevant to your needs. I did not think that was the case here. Nearly all the chapters had some pertinent information, and (while maybe not directly relevant to my current project) were at least interesting to read. I did not feel bogged down with math, most of the explanations made sense and there wasn’t too much needless minutia.

I would definitely recommend this title to anyone interested in video game physics. While there are other books that may be better places to start (as the book sort of assumes you know the basics already) this would not be a bad book to add to the collection. There are actually only a couple game physics books that I *haven’t* read, so it’s getting to the point I will need to stop researching and start writing code. Wish me luck!

Review: Physics for Game Programmers by Grant Palmer

PhysicsForGameProgrammers

Today I will be reviewing Physics for Game Programmers by Grant Palmer, another stepping stone on my quest to build a custom physics engine. Overall I enjoyed reading the book, and I feel like I learned a lot of general things but not enough to base a physics implementation on. Please read on for more details.

What I found most interesting about this text was the explanations of certain aspects of physics that I had not seen covered before. I have already read about 3 or 4 different books all on game physics, and I was expecting this title to be more of a refresher than anything. However, I was surprised to find a lot of things I didn’t know about. In particular, the coverage of drag forces was extremely detailed including things like turbulent and laminar flow and the Reynolds number. Some of the topics covered include: Newtonian mechanics, kinematics, projectiles, collisions, sports simulation (golf, soccer, basketball), cars and motorcycles, boats, airplanes, rockets and missiles, explosions, and lasers. Quite a lot in a little under 500 pages.

There is certainly a breadth of knowledge living inside this book. It was undeniably an interesting read, and I felt like I learned a decent amount. However, I am not sure it really got me any closer to building the physics engine I have set out to create. Let me explain. While there are equations listed in the text, and some example code is given, it is mostly used to support the 2D sample applications. I am not sure there is much I could just pull from the book an paste into a 3D engine. The concepts are sound, and it wouldn’t be a huge stretch to make it work, it’s just not spelled out for you. To be fair, some topics are explained well, like his discussion on differential equations and drag forces among other things.

My main gripe with the book is that it did not really try to explain rigid-body dynamics at all. There are some interesting things talked about, like sports and boats and planes and all that. And certainly there are probably a ton of sports games and simulators that would benefit from that focus. For my purposes, I was looking more for a rigid or soft body solver, and how bodies can interact with each other. Unfortunately, that was not discussed at all.

It’s not that I want to get down on Physics for Game Programmers, and I think Grant Palmer did a great job within the scope of what he was trying to do. The book was entertaining and relevant, it just wasn’t a one-stop-shop for all your physics needs. However, it does cover some basic things well, and includes topics not even touched by some of the other books I’ve read. That alone would make it worth reading, just set your expectations correctly. Once you are ready to make an actual implementation, you will likely need to seek other books or papers. But I guess it is almost always the case as one book can rarely impart all the knowledge you need in any given topic. To sum up: I liked it but wanted more.

Review: Shadow Algorithms Data Miner by Andrew Woo

ShadowAlgorithmsDataMiner

Shadow Algorithms Data Miner by Andrew Woo was an interesting find for me. The oddly titled book was released over 2 years ago, yet there wasn’t a single review on Amazon (where I purchased the e-book). However, I read the table of contents and it seemed pretty extensive in it’s coverage of computer graphic shadow techniques. So I decided to take a chance and purchase the item. I am glad I did.

Prior to vacationing in Japan, I started reading the book and managed to finish it before the plane ride back. It was only about 200 pages long, somewhat short for a technical text, but Woo managed to fit a lot in there. He speaks about planar shadows, shadow depth maps, shadow volumes, ray tracing, hard and soft shadows, filtering techniques, and more. While it certainly was not exhaustive, I don’t think that was Andrew Woo’s intention with this title. It is much more of a broad overview, with ample references to relevant papers and topics to research further on your own. With that in mind, there is almost zero code or pseudo-code contained in these pages. He just explains the general concepts of how the algorithm functions, and it is up to you to implement after further research.

For any readers using Kindle tablets, you will be in for a treat. The Kindle e-book is formatted exactly like the hard-cover, and happens to be about the same aspect ratio as the Fire HDX. This means it looks basically like a real book, everything is formatted perfectly (a major issue with technical e-books), and you don’t have to zoom or pan around. The disadvantage is that you cannot change the font size or background color like you can with most Kindle titles. A fair price to pay for a more streamline experience.

Overall I was impressed with Shadow Algorithms Data Miner and would recommend the book to anyone wanting to learn about computer graphic shadow techniques. Keep in mind the book is highly approachable, but it may be a tad simple for advanced users. If you already have a good understanding of how different shadow algorithms work, you probably won’t find any gems here. On the other hand, if you are starting out this book can be invaluable. It has my seal of approval.

Creating a 3D Game Engine (Part 23)

engine_zero_08

While I implemented frustum culling a little while ago, I never actually coded a proper bounding volume. For the  bounding test I was using a sphere, but I just set the radius to some hard-coded value. This was fine when I just had a bunch of similar sized cubes on screen, however it broke apart once I started getting varied models imported. This week I decided to do something about it.

After a quick Google search for bounding spheres, I ended up on Wikipedia. There was some good information there regarding minimal bounding spheres (also known as smallest enclosing ball or minimal enclosing ball), and I was given a few options for different algorithms. After reading the whole page I become very interested in the “bouncing bubble” algorithm, especially due to the spiffy gif animation showing it in action. The brief explanation seemed easy to follow compared to the other methods, and the error percentage was only around 1-2% which is much lower than Ritter’s algorithm (which can be as high as 20%). So far so good.

Unfortunately I found next to nothing on the internet describing the method. What I did find is a post from last year on StackOverflow by someone in the same situation as me. This lead me to the original paper describing the method, which was for sale for $18 on Grin.com. Honestly I don’t mind paying the money for quality information if it’s going to help improve my engine. The article was fairly easy to follow, and provided just enough information to base the implementation on. At first I was disappointed that no source-code was given, but I managed to pull it together using the pseudo-code. All in all I think I spent about 4 hours on it (including research) so I’m happy to have got it working so quickly.

Below is my bouncing bubble implementation. Keep in mind I have not profiled this code, it’s possible it’s not fully optimized. However, I’m only doing the calculation when a 3D model is loaded, so it shouldn’t effect performance at all really.

BoundSphere RenderCore::calculateBoundSphere(vector<Vertex> vertices){
	Vector3D center = vertices[0].position;
	float radius = 0.0001f;
	Vector3D pos, diff;
	float len, alpha, alphaSq;
	vector<Vertex>::iterator it;
 
	for (int i = 0; i < 3; i++){
		for (it = vertices.begin(); it != vertices.end(); it++){
			pos = it->position;
			diff = pos - center;
			len = diff.length();
			if (len > radius){
				if (i < 2){
					alpha = len / radius;
					alphaSq = alpha * alpha;
					radius = 0.5f * (alpha + 1 / alpha) * radius;
					center = 0.5f * ((1 + 1 / alphaSq) * center + (1 - 1 / alphaSq) * pos);
				} else {
					radius = (radius + len) / 2.0f;
					center = center + ((len - radius) / len * diff);
				}
			}
		}
	}
 
	return BoundSphere(center, radius);
}

Also, I should remind myself to do benchmarks more often. After I added all the normal mapping and shadow stuff performance has taken a big it. Before I was getting likely over 2400fps, now I’m only getting around 1600fps. Still acceptable, but maybe could be better with such a simple scene. The culling definitely helps, but I may have to look into other optimizations like batching and instancing if I want to get the speeds I need for my project. I’ll probably get started with the physics engine aspect soon and then optimize later if I can’t meet my 120fps target. Cheers.

Creating a 3D Game Engine (Part 22)


 

After struggling for a bit with the shadow mapping implementation, I finally have something presentable. I followed a tutorial from Microsoft and thought I understood what was happening. However, it required a lot of changes in the rendering code and it took a little while to get things working. Even once it was somewhat functional, I still had some issues with what they call shadow acne. It seemed really bad on the self-shadowing side of objects. I tried tweaking all the values I could (i.e. the bias in the shader), but I was not able to get it to look right. Finally I just set it so that there wouldn’t be self-shadowing of a single polygon. This happens to look fine, since the lighting equations make the polygon shaded anyhow. I’m not sure if this will work for every single instance, but with the simple models I’m testing with it’s fine.

I’m at the point where I really want to get the physics engine up and running. Still have some more research to do, so I bet it will take some time before I have something more to show. But it’s coming along.

 

Review: Physics for Game Developers: Science, math, and code for realistic effects by David M Bourg & Bryan Bywalec

PhysicsGameDevs

I can say I had a pretty good time with Physics for Game Developers: Science, math, and code for realistic effects by David M Bourg & Bryan Bywalec. It is a very approachable text, and makes a somewhat complex topic more manageable with concrete examples. It’s definitely a book about 3D physics, but many of the samples are done in 2D for clarity. I did not find this to be an issue, and most of the equations and code were digestible. I did see some heat on the reviews, saying that there were errors in some of the code. I could not verify this, but just from reading the book the authors seemed knowledgeable.

At the beginning, each chapter covers a fundamental aspects of physics. Later they go into examples that could be used in a game. Some of the topics covered include: Newton’s laws of motion, kinematics, force, kinetics, collisions, projectiles, particles, rigid-bodies, springs, aircraft, boats, cars, guns, sports. In the last section it goes over some emerging technology and how they work, like touch screens, motion-sensors, 3D displays, and the physics of sound. I very much enjoyed the whole book, and liked that there were practical examples to go along with every topic.

Many technical books can get lost in the math equations, but I found this text to have a good amount of explanation and not too many long proofs. This could be a pro or a con. On one hand it makes reading it a little more straight-forward, but on the other-hand, it may leave out some important details when it comes time to create your own implementation. I’m not sure I would want to base a physics engine solely on this book. It’s not that it was bad, but it was more of a general concept thing than giving you all the code. That said, there is a good amount of code in the book, it just may not be as complete as some other sources.

Overall I was satisfied with Physics for Game Developers. It may not be my favorite physics book (I found Game Physics Engine Development to be a lot better for my purposes), it still seemed worth reading. Now I am beginning to realize the creating a physics engine will not be easy, but the more I read the more I feel capable of this task. I think the book has helped.

3D Engine Dev Log
3D Engine Dev Log