Game Modeling Database

Moved from TranslatorPattern:

This may be a case of GreencoddsTenthRuleOfProgramming. If you have to keep re-writing data structures for different purposes, then perhaps a database is in order. --top

Perhaps in some cases, but not necessarily. TranslatorPattern is justified where a data structure that may be appropriate in one context is not appropriate in another, but both are needed simultaneously. (Note that I wrote "data structure", not "database". A "database" is built from data structures, and typically implies more than just data representation, such as persistence, ad-hoc querying and transactions.) For example, a game may maintain a data structure of game objects to support planning in the AI subsystem, but also needs to render the game objects to a graphical display device. What the graphical rendering engine expects as a data structure may be fundamentally different from what the AI subsystem expects -- which may be different yet again from how the overall gameplay logic expects to manipulate the objects. Use of TranslatorPattern makes it possible to maintain the game objects in one data structure, whilst making it simultaneously appear to be data structures appropriate to gameplay logic, graphical rendering, and AI planning. There is no database system that would efficiently support the manipulations typically performed by gameplay logic, graphical rendering, and AI planning, and the typical facilities of a database system -- persistence and caching, ad-hoc querying, transactions -- would represent needless overhead.

However, a relational DBMS implementation might use the TranslatorPattern to present a variety of different internal data structures (e.g., BTree, hashtable, red-black trees, etc.) as indexes, or to present a variety of data storage arrangements as tables.

I never said it was the best tool in all cases/domains. Of course ItDepends. Maybe in the future a database will be a realistic option for games as they are able to fit and work well on smaller "devices". When you are working with boat-loads of attributes and attribute interactions that need to be "re-projected" for different circumstances, a database grows into a nice option. Possibly related: AreRdbmsSlow.

The issue is not whether devices are mobile or not, or whether you're developing games or not, but whether or not you need database facilities like persistence, transactions, and ad-hoc queries. If you need none of these, there's nothing that stops you from using an implementation of the RelationalModel (or whatever model you like) to represent and manipulate data -- sans persistence, transactions, or ad-hoc queries. It's just not a database nor is it a DBMS. The RelationalModel -- i.e., types, values, relations, tuples, and the RelationalAlgebra -- can be used very effectively to represent and manipulate in-memory data without the overhead of persistence, transactions, or ad-hoc queries.

Games can or could use concurrency and persistence. And I'm not sure what your definition is of "ad-hoc query". DB's are often used for much more than one-off queries. In fact, I'd guess it's less than half of all issued queries.

Games could use persistence for say you want to shut down your computer and come back the next day to play where you left off. Existing games have this, but they are approaching GreencoddsTenthRuleOfProgramming to pull it off, often with awkward "save points". (I'm not a big gamer, so please pardon me if I use the wrong terms.)

And concurrency could be used to coordinate between the AI engine, game "manager/coordinator" engine, and the graphics engine. It's even conceivable to model each object (ship, soldier, tank, etc.) as an independent object with it's own processor space. It could use standard ACID and transactions to communicate/change its state and check on other objects' state. This technology could also be used to scale it up to multi-player gaming, as the distinction between an AI/system-controlled object and an actual player-controlled object can be blurred and/or abstracted away. Full-out gaming potentially could require almost all the data-handling features of a typical CRUD and/or web app. Many business "tracking systems" have a lot of similarities to games in that one is simulating "objects" moving around work-flows, warehouses, user desk, etc., tracking their status, changes, interaction history, etc. Mostly it's just the pace and pretty graphics that are lacking, compared to games. (One tracking system developer made the Delete button trigger an explosion sound on his app. Poor professionalism, perhaps, but many got a kick out of it.)

And rolling your own query language may require a learning curve of the DB user (app-side developers) that may not be necessary. (There are some annoying things about SQL, I agree, but none are arguably sufficiently annoying enough to dump a time-tested and familiar standard to roll your own.)

How many of the typical DB features are necessary or likely in the future before a DB becomes a viable option? Maybe the industry is just stuck in a RAM-centric habit.

And I am not against the idea of a "crippled" DB that disables or excludes features for speed, space, etc. But in theory one should still have the option of switching those features back on and/or upgrading the DB engine to get fuller DB features without rewriting much of the app code.

Games use a variety of mechanisms. The classic "save points" of first-person-shooters, arcade games and the like are more about tradition and simplicity than anything else. Gamers tend to be conservative in their expectations, and expect to find a "save game" facility and associated saved game files that can be trivially moved from machine to machine. Saving and loading game state to a file is, in most cases, no more than a few lines of code. Massively multiplayer online games and social networking games, on the other hand, are typically backed by DBMSs precisely to facilitate data sharing and instant resume without any notion of "saved games".

Concurrency controls are certainly used to coordinate concurrent subsystems, but DBMSs are not the only way -- or necessarily the right way, depending on requirements -- to control concurrency. Furthermore, in many cases the SQL coding (what I presume you mean by "all the data-handling features") of a "typical CRUD and/or web app" is needless overhead and effort. If you need to iterate data structure p, there's no reason to issue 'dbConn = DBMS.connect("mydb", "me", "secretpwd"); resultSet = dbConn.query("SELECT * FROM p"); Iterator i = resultSet.iterator();', or even just 'Iterator i = iterate("SELECT * FROM p")', when all you want is 'Iterator i = p.iterator()'. This is the case for much of game development. That doesn't mean there aren't game UseCases that demand the facilities of a SQL or relational DBMS -- there certainly are -- but most game development doesn't require it and wouldn't benefit from it. For example, most of the time, there is no need for a query language at all. For the rare times you need to retrieve a subset of the objects in a container, an iterator that supports a predicate expression in the native language is both sufficient and TheSimplestThingThatCouldPossiblyWork.

I'm sure there are naive game developers who decry and avoid DBMS features for all the wrong reasons, and thereby make more work for themselves when they actually need a DBMS. There are Web and business application developers who do that too. However, most successful game developers are experienced and rational programmers with a broad knowledge of SoftwareEngineering, including databases. You need broad knowledge to develop games, which often incorporate graphics, resource management, event handling, concurrency and multitasking, UI interaction, high-performance I/O, physics modelling, AI, state management, and -- sometimes -- databases. A good (game) developer has a healthy dose of pragmatism -- he/she knows to use DBMSs when DBMSs are warranted, and knows to use other things when they are not. A DBMS is the right tool for some jobs, but it's not the right tool for every job. PickTheRightToolForTheJob.

One can make a pretty compact way to iterate over something with a plane-jain table query if the SQL pattern is always the same, but the power/flexibility comes with the ability to filter, sort, and/or join as needed. I'd be surprised if non-trivial games could get by with mostly using simple query equivalents. Perhaps WaterbedTheory is at play in that for YOUR style of software design, you simplify queries/iterations at the expense of something else being more complex. Without seeing the requirements, design, etc. I cannot make that determination here and now. And most game developers probably don't use a database because the machine resources and latency profile are not yet a good fit in most cases. Thus, I don't think it's mostly about the capability of databases from a software design standpoint, but rather machine-centric issues at this point in history. Plus, the tradition/history of existing game libraries and conventions is probably RAM-centric in collection handling for prior-era performance reasons, and there may be a mini QwertySyndrome hump preventing a larger set of DB-oriented libraries and conventions.

I can envision cases where objects "remember" encounters with other objects and learn and/or alter their behaviors based on that. And each object could "ponder" future behavior by computing "action trees" to test strategies/plans. And also allow a category "set" sub-engine that allows objects to inherent traits based on membership to multiple arbitrary categories. One wouldn't have to hard-wire such traits/membership up front even: trolls can become kings and kings can become trolls and do troll things and king things. (And anti-categories that prevent a given object from being members of selected categories.) You start to get a lot of many-to-many relationships with all these, something that RAM-centric techniques will likely have difficulty with. You may not even have to release lots of "new" games, as one game engine can be almost anything via setting DB attributes alone. Maybe new "versions" get prettier graphics, but the rest is just fancy usage of DB attributes. It's kind of like Cyc: you don't release different incarnations, you add to it. (And then it grows into The Borg and eats humanity. Or the Matrix. Maybe we are in a DB-based game and don't know it. "God" is the table-head who built the bugger.) -t

A game "programmer" should mostly be adjusting attributes such as categories, weightings, fade durations, probabilities, personality tendencies (aggressiveness, honesty, friendliness to strangers, friendliness to own kind, etc.), etc. Actual app-code programming should be things like making a fancy nuclear rainbow explosion with flare-like remainders, but the conditions for when and where this fancy dancey explosion is triggered should come mostly from attributes, not app code. The "programmer" would largely be filling in CRUD-esque screens. (Although one could perhaps divide the labor into "worlds/location manager", "character/species manager", and visual programmer.)

And if you depend on so many attributes and associations, then you'll probably need "reporting tools" to sift, sort, search, filter, aggregate, etc. all those attributes using combinations of report writers, ad-hoc queries, QueryByExample, etc. -t

Successful game developers tend to be broadly-experienced SoftwareEngineers with no particular biases except toward getting games to market. If a DBMS helps that, they'll use a DBMS. If it doesn't, they won't.

That, of course, is a sound strategy for SoftwareDevelopment in general.

For example, the A* algorithm (see http://en.wikipedia.org/wiki/A*_search_algorithm) is commonly used in games for navigation, for finding an efficient set of game moves against an opponent, for finding an efficient set of traversals between two nodes in a graph, etc. Whilst you could conceivably implement it using a DBMS, why would you do so? It's simpler to code it natively assuming you have standard containers like sets and lists -- which is the case in any modern programming language -- and neither persistence nor transactions are required.

There is or should be more to game development than mere path efficiency optimization. Maybe too many games are stiff and predictable because they don't go attribute-based. And use of a database and use of A* are not mutually exclusive.

Sounds like you're describing influence mapping. See http://aigamedev.com/open/tutorial/influence-map-mechanics/ Please don't assume that because you're unfamiliar with game development that game development is an immature, superficial, or narrow-minded field.

A sufficiently complex "influence map", along with world/tribe/beings/ship/military attribute tracking is going to be sufficiently complex such as to trigger GreencoddsTenthRuleOfProgramming. The interaction between all of these, including studying the interaction for debugging and tuning, is too complex to manage well via self-rolled data structures. And again, only recently has the hardware caught up in order to make DB a viable option. The habits and tradition of the limited-hardware past may take a while to move past because those parts are road-tested, mature, and familiar to developers.

Can you demonstrate how an influence map "is too complex to manage well via self-rolled data structures"? It sounds like you're speculating.

Maybe I'm thinking too big here. I'm envisioning a kind of Star Trek-like universe where there are hundreds or even thousands of planets to visit and species/villains/friends to encounter (intentionally or unintentionally), all with unique characteristics. And they could even be dynamically created and/or evolve over time, either in a fractal way, and/or as one explores the edges of territories. It's sort of Trek and LOR meets The Sims. One is not just jumping over and blowing up obstacles, but also factoring in the motivations and personalities of friends and enemies to achieve goals and survive, who come and go and are re-encountered later on. If you limit the scope of the game, then hard-wired "stuff" may be GoodEnough.

I thought the above was about game development in general, not a particular game that you have in mind.

As I've pointed out twice, there's nothing that precludes using databases where databases are appropriate, and they do get used where appropriate. They are not used where they are not needed. I'm not aware of any game development project that needed a database and ruled out using one because of some "tradition of the limited-hardware past", or any other unreasonable reason. It's certainly possible for it to happen, but not because there's any cultural bias against databases (or any other technology) in game development. Of course, there isn't any particular bias toward databases (or any other technology) in game development, either. The biases that once held sway (e.g., C++ no matter what -- and before that C no matter what, and before that assembly language no matter what) have almost entirely dissipated under the force of competition and natural maturing of the field.

Maybe the maturing ain't done yet. The history of system design went from machine-friendly languages, to more human-friendly languages, and then databases came into play for larger and more complex attribute management. Gaming may be on a similar progression pattern.

Maturing is a process rather than a destination, but once again, there doesn't seem to be any evidence of general unreasonable bias against databases in game development at its current state of maturation. Databases tend to be used where appropriate, and not used where appropriate. Nor is this particular to games development -- it's generally true of software development.

Rather than talk in generalities, maybe we can select a particular game or "kind" of game and analyze it in terms of managing the complexity of attributes, categories, changes, associations, etc.

How about a popular component of game development that gets used in a variety of contexts in a variety of games? I suggest analysing the A* algorithm, which I mentioned above. I have a simple, generic example of A* written in Java, which I use to illustrate various SoftwareEngineering issues to game development and ComputerScience students. You could use it to demonstrate how A* could be implemented -- and perhaps improved -- using a database. It should be a relatively easy task, because my implementation has been designed to allow replacing components of core functionality to facilitate illustrating different implementation strategies. Sound good?

Path-finding optimization is only one part of possible game-play. Databases are about integrating and managing the relationships and attributes of entities. Integrating a database with A* is certainly something that is worth analyzing, but it's kind of a side topic or sub-topic to the general question of when, where, why to use a database for games. (It is part of "how", but not the only part.)

Of course, but path-finding is so universal and easy-to-understand that it makes for an ideal test case. It's certainly of far more reasonable scope than writing an entire game, and writing an entire game inevitably begs questions about whether the particular genre chosen was particularly suited to using a database, or particularly suited to not using a database. Since A* is independent of genre, that argument doesn't enter into it.

Re: "...is needless overhead and effort. If you need to iterate data structure p, there's no reason to issue 'dbConn = DBMS.connect("mydb", "me", "secretpwd"); resultSet = dbConn.query("SELECT * FROM p"); Iterator i = resultSet.iterator();', or even just 'Iterator i = iterate("SELECT * FROM p")', when all you want is 'Iterator i = p.iterator()'. " 

 function entityIterator(entityName, _
   optional colSelector="*", _
   optional whereClause="(1=1)", _
   optional orderBy=""
   optional connection=app.defaultDBConn)
   as Iterator  // returns type Iterator


   var useSql, dbConnm, resultSet


   useSql = "SELECT " & colSelector & " FROM " & entityName & " WHERE " & whereClause
   if Not isBlank(orderBy) then
       useSql.appendString(" ORDER BY " & orderBy)
   end if
   resultSet = connection.query(useSql)
   return resultSet.iterator()
 end if
 //(Pseudo-code roughly based on VB.net for its named parameters)
Thus, if you only want the bare minimum, you could: 
  Iterator i = entityIterator("p")
but one could use optional parameters to get custom column selection, row filtering, and change the default sorting. You can have short and you can have the usual table handling (except joins).

But that's my point: If all you need is Iterator i = p.iterator(), then you don't need to write the above entityIterator() function. The above presumes that custom column selection, row filtering, and changing the default sorting are something that will need to be done frequently. Generally, they aren't. When they do need to be done, there are container-oriented ways of doing them that are easier to express in the native language. For example, "column selection" is nothing more than referencing the attributes or methods that you need to reference, row filtering is often done using predicate functions, and sorting is accomplished by using multiple containers that each maintain a desired sort order. All of these are computationally less overhead (and usually syntactically less overhead) than constructing SQL, even if wrapped in something like entityIterator().

Where SQL is beneficial is when column selection, row filtering, and changing ordering are the very essence of the manipulations you typically need to do. Business applications are a classic example. Another place where SQL is beneficial is when column selection, row filtering, and changing ordering are driven by the user on an ad-hoc basis, such as user-defined reports. There are certainly aspects of certain games that benefit from both of these things, and databases do get used to implement them. However, typical game AI algorithms (planning, path-finding, finite-state machines, plus some neural networks, genetic algorithms, etc.) don't benefit from SQL -- it would be more awkward to bend the implementations to suit SQL than to implement them in a typical imperative or functional language -- and typical 2D or 3D rendering engines don't benefit from SQL, as they're mainly bit-blitting and vector and matrix operations, plus some trigonometry.

"Computationally less overhead" perhaps may be true. But there are times you'd rather let the machine slave away harder to simplify software and modeling creation, management, and debugging. There will likely be entities resembling characters, species, worlds, cities/areas, ships, weapons, battles, and lots of associations between all of these, along with categories.

If we're talking about implementing game logic, game AI, game physics, and graphics rendering -- i.e., the majority of the game engine side -- then it's both computationally less overhead and, generally, syntactically less overhead to use containers instead of a database. For modelling the game "world" that you've described, with species, weapons, cities, etc. -- which sounds like a typical RPG-genre game -- certainly databases are appropriate. Currently, SQLite is popular for embedding database functionality in games.

"Game logic" is rather open-ended. And if you can get the DB to do more of the work, it's not "more syntactical overhead". One just has to change their thinking toward programming via attributes instead of programming via functions.

"Game logic" is only one aspect of game development. There's game AI, game physics, UI, and graphics rendering, too.

I invite you again to demonstrate how you can "get the DB to do more of the work" using an implementation of the ubiquitous A* algorithm. Interested?

No. A* is relatively boring. I'm far more interested in "business logic" than optimization. And it probably wouldn't tell us anything, for I do not propose using a DB for JUST things that A* concerns itself with. It's not a good test. A reasonably complex game that attempts to model social and political interaction and has a lot of potential dynamic characters is going to have a fair amount of entities and associations between those entities. The AI engines and physics engines and graphics engines will perhaps be better off if they use the same "collection handling" engine that the other entities do, in part so that JOIN's can be done, and in part for consistency of game engine conventions. If there is a repetitious collection handling pattern(s) that you want to put app-side wrappers around to make simpler, you can go ahead and do that. The existence of a database does not preclude app factoring. And yes, in some cases hooking such into the DB may create more machine overhead. I get that. But powerful tools often do tax the machine more.

I think what you really want is integration of the database with the app language such that the issue is not really about database versus app collection handling, but about linguistic integration.

I've got that. I'm the leader of the RelProject, which integrates the database into the application language and vice versa.

However, I'm not convinced that integrated RelVars, relations, arrays and tuples -- plus a powerful query language -- are an effective substitute for conventional data structures like stacks, lists, queues, trees and graphs. I think you need integrated RelVars, relations, arrays and tuples -- and the canonical data structures as built-in containers -- plus a powerful query language. If all your algorithms need only sets, then just relations and their associated mechanisms are sufficient. However, if your algorithms rely on lists, queues, trees, graphs or stacks -- and many do -- it requires effort (and awkwardness) to substitute lists, queues, trees, graphs or stacks with relations.

So, whilst you may find A* "relatively boring", it is a recognised algorithm that can help to highlight -- along with other well-known algorithms -- the distinction between a database-oriented approach and a conventional data structures approach. Whilst modelling "social and political interaction [with] a lot of potential dynamic characters" may be an interesting modelling problem, it's also one for which a database approach is obvious and uncontroversial. I don't think there's anything new to learn there, whereas attempting to implement various algorithms using a database approach instead of conventional data structures may lead to interesting discoveries.

If I am understanding you, you are saying that within the current "library of knowledge", the popular and useful algorithms are all defined in terms of "traditional" data structures, and thus a translation layer is required to adapt them to RDBMS usage. Therefore, what's really needed is a relational version(s) of such algorithms. That may be true, but it's not something I'm currently interested in delving into. (And I agree that RDBMS could use some tree- and graph-friendly operators/syntax/etc., but perhaps as library additions rather than hard-wired into the "root".)

Yes, popular and useful algorithms are defined in terms of "traditional" data structures, and relational versions of such algorithms are particularly what I'm interested in. Finding them would make it reasonable for general-purpose languages with integrated implementations of the RelationalModel (like the RelProject) to dispense with built-in stacks, queues, lists, trees, graphs, etc., and it would permit dispensing with the means to create your own linked data structures via pointers or references.

EditText of this page (last edited July 29, 2014) or FindPage with title or text search