Twenty Dollar Wiki Device
Twenty Dollar Wiki DeviceChallenge: Build a Wiki Device with a parts cost of no more than twenty dollars which is usable as a portable wiki. Include feature to make it compatible with a desktop or laptop computer's PersonalWiki, with a connector allowing this use. (Serial port regular or USB).
I've got an 8051-based microcontroller (8bit Harvard with 64K program flash and up to 64K external data RAM) that I'm using to improve my embedded systems skills. I'm planning on adding some external flash memory and RAM to it. I can talk to the micro from any serial port, so I have all the hardware I need to build a PersonalWiki. It's probably not going to be very useful but I'll learn a lot of useful stuff building it. Here are some of the problems I'm up against, suggestions are welcome:
It will be written in C, with some 8051 assembly. On most micros, C is the high-level language with the most support. There's some Forth and Basic but after that it gets really spotty.
Guys, thanks for the discussion below. I'll leave that in thread mode for now and distil learnings here:
Don't forget that the stack size issues need to be investigated for C, too, not just Forth.
I think if you're willing to absorb the learning curve, ForthLanguage would be good for this. Once the Flash read/write primitives are in place, you can store pages as blocks (with your choice of block sizes) without having to worry about file system issues. And, because Forth supports LRU and write-only-when-dirty concepts, you can get decent write performance.
Forth will give you the greatest bang for your buck, as you can fit more functionality into the same RAM. You also have dynamic extensibility available if you reserve some page/block space for new code words.
I wrote an etch-a-sketch using graphic characters on a text-mode terminal in 48 lines of Forth, and the Forth kernel was (including the editor and assembler) only 16kB running on a Z80. The drawings were saved as simple vector maps on pages (blocks) of their own, using a simple compass point notation, pen-up, pen-down, and so on.
You can cram all kinds of things into a small Forth system. Yes, CeeLanguage "has more support" but you're not looking for more support, unless you're hiring a team to maintain this. We have Forth guys here on the list who, I'm sure, would be happy to help out.
As a compiler guy, I must say that there is a huge impedance mismatch between C and processors like the 8051 or even more general purpose 8 bit CPUs like the 8080 or z80, whereas Forth is likely to make much more efficient use of the processor and available RAM.
With 64K, you've got some space to throw away as long as you don't go wild, so C might work out ok, but it would burn through RAM like a bat out of hell and likely be slower than Forth, too.
My experience is dated on these subjects, but I'm not aware of anything relevant that has changed over the years. This is the niche that Forth was designed for, basically, whereas not so for C.
As I recall, the 8051 takes something like 12 clock cycles to execute a NOP and 20 clock cycles to actually do a real instruction, so you're starting with a slow processor in the first place, and C will add its own speed overhead in addition to space overhead.
I know Forth without being an expert in programming in it (implementing a language isn't the same as knowing its idioms reflexively), but my take on it is (I think Leo Brodie said this) that although Forth has low level syntax, nonetheless in certain ways it has high level semantics, so I don't think this would be a huge sacrifice, either. Forth code can be exceedingly elegant.
-- DougMerritt
I don't see a problem with using C for small/embedded devices. It worked fine on my old 8086(with 8087 math co-processor, which added an extra 2 hz - now that's speed!), heck it even worked on my Apple ][+(with 16k expansion card!!!). Works even better on my old 386 laptop using TurboC(free from borland). Also works cross-platform with straight C win32 code when ported to the XScale(ie.PocketPC). Admittedly your memory constraints are higher, but tight C code is usually on par with mediocre assembler. -- LayneThomas
No no no. The 8086 is 16 bit and has a very rich instruction and register set compared with the 8051, which isn't even a microprocessor, it's a microcontroller. There's an enormous difference. No comparison at all.
I used C extensively on the pitiful old z80, and it "worked" in some sense, except that I ran out of RAM all the time even with small programs, but when I looked at the emitted code to see why things were so slow, I almost had a heart attack. And the 8051 is significantly worse. The z80 is a paradise compared with the 8051 (and the 8086 is a paradise compared with the z80).
Even if there were some basis for comparison, you didn't say for what values of "worked" it worked. :-)
-- DougMerritt
I might be wrong, though; a commercial company says "The C51 Compiler allows you to write 8051 microcontroller applications in C that have the efficiency and speed of assembly language". Probably just marketing nonsense, but maybe not, maybe they're telling the truth. I don't quite see how, but...
-- DougMerritt
Doug, what makes Forth better in this environment? Does the stack allow you to avoid using expensive 16 bit pointers everywhere?
Partially. Efficient C makes heavy use of simultaneously active pointers in almost all applications, and small processors just don't have the registers to support this, so you spend most of your time screwing around with loading and storing pointers with awkward series of instructions. And the overhead of doing standard function calls and parameter passing chews up an amazing number of instructions.
Forth isn't perfect in this regard, but there's primarily just the one pointer, to the parameter stack, plus not infrequent reference to the return stack used as a second data stack. It's not as bad as C on average in that regard, and usually will directly borrow the native function call mechanism, and there isn't an additional mechanism for parameters; it's the same mechanism as for working data.
Best of all, though, it is extremely natural in Forth to extend the language with new "words" written in assembler that use some idiomatic sequence that is natural for the cpu, but is wrapped in a reasonably abstract way so that it is natural to use as well. This sounds similar to rewriting an inner-loop function in C in assembler, but it isn't really - it works out to be much more natural to do this in Forth. So you end up with a system that is higher level than assembly but has, by and large, the same speed advantage, and sometimes is even smaller in its RAM usage than assembly.
-- DougMerritt
I guess the parallels would be similar to nVidia's shader language Cg - which isn't really C, more of a way to compile register combiners as if it were C(it doesn't even have loop, but it does have extremely fast trig operations). With a little more generalized processor, and a C compiler that maps closely to the assembler it could be good enough to write reasonably complex programs. On the other hand I'm not a microcontroller guy. I'm sort of bothered that the 8086 was a paradise, especially writing this on something so relatively more complex. -- LayneThomas
Well, just remember to compare it with 6502 CPUs. In addition to the 64K memory address space, there are only two (8-bit) registers, IIRC. And the instruction set is much more restricted than even the Z80.
Remember, the 8086 is as big a jump up from the 8080/Z80 as the Itanium is up from later Pentiums. -- EuanMee
On C and microcontrollers vs microprocessors... I have to agree with Doug. I did a very timing-sensitive project on a Atmel 90S1200 and ended up just giving up and dropping to assembly. Forth was not an option, in this instance.
[OTI did an embedded Smalltalk for Tektronix many years ago, that ran on very limited hardware. The Smalltalk kernel and VM can get pretty small. Some of the Tek alumni here can shed some light. My understanding is that development was done on a regular workstation (I suspect using Envy/Developer, another OTI product) and then "packaged" into an executable that was then burned into ROM on the embedded system. Tek was supporting realtime special-purpose oscilloscopes. I suspect that swiki or something similar would not be a stretch. The squeak stuff is all open-source, including the VM code. Lots of us wrote at least portions of the Smalltalk VM in assembler, and it isn't too hard. Like Forth, its a stack-machine. Much or most of the really hard stuff is up in the image anyway, making the VM implementation easier. Also, why limit yourself to 64K RAM, is that a 8051 constraint? All I'm saying is that Smalltalk community burned lots of midnight oil making Smalltalk work well in embedded systems, and much of that work is available for the asking - and there is a very good impedance match between Smalltalk and Wiki.]
Most 8051s are limited to 64K. There's a 16bit address bus although I've heard of 8051 derivatives that have much larger address spaces. Most people switch to more powerful 16 or 32 bit micros for larger projects. Smalltalk on 8051 is an interesting challenge in its own right but it's a different course. Right now I'm wondering what it would take to implement a Wiki-like thing without an underlying OS or VM. I guess the same could be said, to a lesser degree, about using Forth. -- AndrewQueisser
http://www.zetetics.com/camel/camel51.html http://www.earthlink.net/~forth ftp://asterix.inescn.pt/pub/forth/8051
"why limit yourself to 64K RAM, is that a 8051 constraint?"
For starters, that's the nominal absolute limit for any 16 bit architecture; 2 to the 16 equals 64K, after all. (Harvard architecture splits instruction and data address spaces, so each is 64K apiece, but that's a side issue.)
You guys who aren't familiar with the difference between microcontrollers and microprocessors need to simmer down and stop thinking that they're pretty much the same thing.
The general thing to expect on a microcontroller is 64 BYTES of RAM, instructions in read-only memory burned in permanently in the factory, many missing instructions, instructions that do happen to be there are really quirky, one accumulator register (all arithmetic results go there), one memory address register, a call mechanism that uses a stack builtin to the CPU that is only 4 to 8 levels deep (so that you absolutely cannot nest calls deeper than that, and there is no workaround), a clock speed of 1 Mhz but every instruction requires 20 clock cycles, so that you'll execute 50k instructions per second, etc.
On the plus side they usually have some builtin features for doing I/O to control peripherals such as serial ports (thus the term microcontroller).
Now, it's true that, over the years, some microcontrollers have had extra features stuck onto them, so that a few of them are approaching full-fledged microprocessor status, but this is not the nominal case by any means. Microcontrollers are often used in products where it is important to keep the cost down to five cents apiece. To have as much as 64K bytes of RAM on a microcontroller is a huge luxury, and very non-typical.
You can't generalize your experience with even old slow microprocessors to microcontrollers without looking into it further. Most often they are far more similar to a serial IO FIFO with a few programmable features added. A minimal amount of programmability does not equate to general purpose.
The 8051 family is moderately powerful as microcontrollers go, but it is only a microcontroller. The original was the 8048, which was mask-programmed only at the factory. The 8051 allowing instructions to reside in RAM is already a big step upwards. The architecture was introduced in the stone age... circa 1980 I think.
-- DougMerritt
Makes sense. . . also makes it a terrible choice for a Wiki. . . but a great learning experience -- LayneThomas
Aw, come on, Doug. This is what makes it FUN! Just think of the hacks we can do. My first programming assignment, in my first programming course (in 1970) was a FORTRAN/WATFIV simulation of the PDP8. The instructor handed us the PDP8 handbook, told us that the IO instructions were already written, and began to explain binary algebra to us freshmen. For the final, our simulator had run a PDP8 program supplied by the instructor (that we'd never seen before). We had a PDP8 in the lab that we could use to check our results along - simulate the 8051, then figure out how to bring wiki up on the simulator (I still think Smalltalk would be fun, but as Andrew points out, that's a different course), then burn it. Seems like we could really have some fun with those peripheral controllers and serial ports.
Certainly. And I'm sure he could get a wiki working on this thing even in C, and have tons of fun on it.
That's not what I was commenting on. The point is that several people were speaking as if the challenge were no different than, e.g., an 8086. Why limit it to 64K... why, when we were kids, we'd have given anything for a K! :-)
I can't believe that they had the nerve to run an intro programming class like that. Was there a 99% dropout rate? I'm sure it was fun for those who completed it, though. Oh wait, 1970...this was a graduate class for math grad students.
It was either 1970 or 1971, "Intro to Programming", at CMU - freshman or sophomore (I don't remember which). I think it might have been second semester, freshman year. I know that it was my very first exposure to the concept of computing. I got very good at Fortran, and by the end of the semester I was a VERY GOOD pdp8 programmer. I've always loved virtual machines, I think because of that course.
we'd have given anything for a K!
There's an EwDijkstra paper (EWD1303-0 http://www.cs.utexas.edu/users/EWD/ewd13xx/EWD1303.PDF) wherein he talks about working on an automatic typewriter. The main thrust is to discuss concurrency. One of the side notes dealt with how the price of tubes made the obvious solution of an extra register far too expensive, and how his next project had a budget for the sheer luxury of an extra 6 bit register. Extra K indeed, pshaw...
Well, of course, we had it tough! We used to have to get up out of the shoebox in the middle of the night, and lick the road clean with our tongues!
You try and tell the young people of today that, and they just won't believe you.
There are days when I wish I had the "sheer luxury" of an extra 6 bit register on my PC.
Reference Doug's remarks above about the dearth of memory for microcontrollers: today, in real life, at a salaried job, for a non-insignificant manufacturer of highly interactive gadgets (currently number 2 in their industry space), I write code for 80c751 derivatives of the 8051. This chip sports a whopping 64 bytes of RAM, some of which doubles as register space. There's a roomy 2 KB of code space (we use the big ones), and it's really zippy at 12 MHz. It is an OTP device (OTP = OneTimeProgrammable?), meaning that if you screw up the code, you throw the chip away - they don't recycle like EPROMs.
We use them to manage card readers, keypads, VFD (VacuumFluorescentDisplay?) units, relay banks, and so on. You could use a chip like the 80186 or Z80 or even a larger MC like the Z8 or 8051, but unit cost is a big deal, so we get really clever with small chip resources.
And, yes, we do all the code for this little gem in assembly. When I code for the bigger 8051 I have the luxury of coding in C (with excursions into assembly). We've even got one gadget that uses a 4 megabit (bank-switched) flash EPROM for program space. Huge. Really splurged on that one.
By comparison, our next-gen combo gadget is based on the PXA255 (think "plastic pentium") and will have megabytes of RAM and flash storage. The price jumps from $50 per unit to $600 per unit (did I mention the 1/2 VGA touch screen?), but marketing thinks they can justify it. Oh, and we move from effectively RealTime event handling to ... WinCe! The damned thing is thousands of times bigger and an order of magnitude faster (400 MHz), and we can't do RealTime with it.
The "nuts and bolts" engineers just shake their heads while muttering "RealTime CE ... no, no, no, no ..."
Andrew, I'm pretty sure you can get it done with an 8051 (try one of the 24 or 40 MHz parts) and some flash storage. A serial ANSI terminal interface should be fine. In fact, using Forth, you would actually be able to make it a programmable wiki.
For what it's worth, there are 8051 drivers out there for USB (you would need a USB controller though), so that's another interface approach. I personally favor good old 3-wire serial @ 9600 bps.
I'd really love to see what you wind up with. -- GarryHamilton
What's the max transfer rate on an 8051-controlled USB? Surely it can't approach max USB 2.0 rates, nor even USB 1.0 rates, come to think of it.
Perhaps with a really cheap keyboard and simple text b/w lcd screen, it could be a key-chain PersonalWiki that does nothing else.
It's not really cheap ($400), but a nice Palm-based device (160x560 screen) with a nice keyboard is available at AlphaSmart?. I'm not affiliated with the company, but am intrigued by the device, especially for the education market. http://www.alphasmart.com
Update (20050123) Not quite a twenty dollar device (about $200) According to the video it is nearly indestructible and runs nearly 500 hours on 3 AA batteries. Take a look at what it looks like via a .wmv file.
How's this getting on - any news?
Yeah, very slow progress. I'm tied up at work and have two little kids so there's not much time to work at home. I made a little progress though:
Sounds like a fun project. Serial I/O is clearly the best way to start. Later it might be fun to build a "computer in a mouse" (the mouse cable plugs into a TV or computer monitor), such as
"a simple way to check which pages exist (page table or file directory). Probably some kind of simple hash table." I suspect the SimplestThingThatCouldPossiblyWork is to force every wiki page to start at the beginning of a flash sector. The first couple of bytes of each sector would indicate whether this is a new wiki page (followed by the name of the page), or merely the continuation of the previous page. To see if a page with a given name exists, search through the first few bytes of every flash sector. Reading an average of (guessing) 4 bytes per (guessing) 1 KB sector out of 64 KB Flash, that's only reading 256 bytes. Even at a mere 1 MHz you should be able to do that a few dozen times before the user can blink. -- DavidCary
Couple low-cost ideas for prototype:
Running across this page made be think of a device that I ran into recently, the Ben NanoNote. Not quite $20, but it is a full embedded Linux-based computer, with the capability of installing packages. The screen is probably not sufficiently capable of running a graphical web browser (320x240 resolution), but it looks like a very pocketable device, and packages exist for programs like vim and means, which have wiki capability. Just another little morsel for thought; it definitely has me intrigued... http://en.qi-hardware.com/wiki/Ben_NanoNote --EvanDeaubl
EditText of this page
(last edited August 22, 2011)
or FindPage with title or text search