Time to stop relaxing

September 12th, 2007 by taleks

About half of month I was doing nothing (actually vacations were planned, but they shifted once and shifted totally away at second time :)) I gave my second testing computer to brother, so I’m waiting when ordered notebook will come to doors of my home to continue developing.

There are several major tasks to perform:

1. Check why gziped version of modules are not loaded correctly (rc and forth scripts  e.g. loaded correctly). I’ve unwinded call stack from interact() to working with file systems, it seems there is no nothing criminal, but non-compressed versions of modules are not loaded after first 512 bytes read by file_read(). May be I’ve made somewhere incorrect define of macro.

2. To implement directory listing (parse html result of server-reply). Main purpose to use same code for different listing types (detect numbers anf filenames between tags without actual parsing of html). After first view – this task seems solvable.

3. To comment more accurately commits, that were changing libstand. May be I’ll add also another function there strnstr(), which behaviour is “emulated” now in not very good fashion in pxe_http module with usage of strstr() and manual adding of zero byte at end of readed from socket data.

4. Work a little bit with assembler source. There is now unused code here and there is working, but may be not very correct ISR here. So, change it. Also, PXE call trap is duplicated – so there may be small optimization.

5. Added to loader functions must be revised (some of them really useless, after testing is completed). Comment them.

6. Write more solid documentation. And upgrade existing in some parts.

I hope, I’ll start working at the end of week. After completing of this tasks earlier mentioned telnet client module will be started (may be this will need to make PXE sockets API more similar to common sockets, then it’ll be possible to use common telnet client code, but without advanced terminal settings and signals support).

Successfully tested

August 19th, 2007 by taleks

Since last post were added  loading of RAM-drive, using httpfs precaching and some other small updates and bugfixes, most of time was spent  for testing.

Now rootpath option is used to specify location of root on web server. Also, I’ve found that bootp() doesn’t sets nameip (it was not usually needed for NFS loader, it’s enough to specify IP of server and resolving is not required. But in case of web0servers, which may use virtual servers for different domains for same IP – it’s needed to know domain name, and nameserver option is needed) and thus gives problems for code to work, so by default own DHCP client is used. If  not domain name but IP is specified in rootpath, then bootp() may be used.

Loading of RAM-drive is optional, but was needed to test correct working of code. It’s done simply by adding few  lines to loader.rc script. Actually, it’s possible use for example NFS after loading as root file system (by setting appropriate environment variables), but as long as pxe_http library code must be a little bit independent – best choice was to use RAM-drive (or additionaly to write kernel module for support httpfs. Last variant needed more time and knowledge, so it temporarily was thrown away).

RAM drive image was got from boot floppy image (mfsroot.gz). It contains needed for system installation files. So, after loading by pxe_http kernel+RAM drive  it’s possible to install FreeBSD by choosing ftp as media source.

While testing code in LAN I’ve found that loader trys to get file by small portions (maximum 4096 bytes per request). It is not very optimal, so about month ago I was thinking about simple caching mechanism and now it works. For this purpose, httpfs need access to socket receive buffer (this is done to reduce memory usage, so cached information is actually stored in socket buffer, main problem is in removing from buffer header and watching when cached data is empty). Such ability breaks abstraction of code layers, but raises up connection speed and reduces load of http server. While testing it lowered downloading time from about one minute to 25-30 seconds. After setting lower waiting time in pxe_await.h, code response to incoming packets become higher, and downloading is performed in 15-25 seconds at LAN.

After caching was checked, I’ve started testing of code with remote server, that is really remote and situated in 12 hops from my home computer. Here was first surprise, TCP code was working really strange: some packets were never received. During debugging also bug with resending was fixed (segments were waiting to be acked bigger aequential number, then needed).

So, packets were not receiving and code not worked in WAN. I’ve spent many hours trying to make it work. Ed (my mentor) said simple idea – to tcpdump on server side. Well, I don’t know why this idea have not appeared in my mind itself. tcpdump showed, that all is ok – packets from me correctly are coming to server, and from server to me are also sent, however not acked. Debugging of pxe_core showed that they are not reaching my NIC. I’ve started comparing two dumps and found rather interesting thing: only packets with segment size fuly filled to 1460 (default TCP MSS in pxe_http) were not transmitted correctly. I’ve changed MSS to 1260 (10 hops * 20 bytes is about 200 bytes…) and code started working correctly. Packets were sent with Don’t fragment IP flag, so I think somewhere on the path to me router dropped them cause couldn’t sent it not fragmented. May be it’s just my ISP side problem.

After getting code working correctly in Internet, I’ve tested it on two different web servers (first uses Apache, second one – nginx), both worked correctly. And to make sure code is really working, I’ve managed my friend to try it from his home through ADSL modem (at my home, Internet is got through VPN via ISP Ethernet based network, so router at my home is second computer with pppoe client). It also worked.

Last check was to test again NFS loader after some changes in pxe_open() function. It also worked, so finally code seems ready to be tested widely and tarball of current version of code will be sent as result of GSoC.

Finally, I’ve wrote simple documentation,  that may be useful for setting pxe_http to work or use it API. Well, it’s not covers every function of pxe_http code, but gives main information, needed to work with it.

Next two weeks I’m planning get more information about ipv6, spend summer time and do nothing usefull at all, except fixing any simple bugs found in code. Later I’ll  try to add ipv6 support for pxe_http and may be implement some other ideas.

Boot-reboot..

August 1st, 2007 by taleks

Finally I’ve booted kernel on real machine without unexpected rebooting it in process of booting.

While testing code, I’ve tried smaller buffers (16, 8, 4, and 2Kbytes were used) and tests figured out, that code with 4 and 2 kbytes is able to boot kernel on real machine, bigger buffers are working on virtual machines. Keeping in mind fact that memory allocation/freeing is done most probably correctly (it’s done by libstand functions, that are trusted) it was difficult to understand why smaller usage of memory for buffers gave such strange results.

First investigation showed, that loader’s heap is allocated over PXE data (in base memory higher 8d000h). That was fixed (higher bound of heap was limited to address from which PXE data starts), but hadn’t solved the problem. Yeah, sad moment, I had faith that problem is in heap allocation. Next suggestion was that pxeboot is rather big binary. In fact, PXE specs recommend to make download of boot images in two steps: remote.0 and remote.1. Remote.0 placed at well-known 7c00h:0000 and must not exceed 32Kbytes (cause “32K size limit provides the advantage of being able to clean up and fail gracefully” (c) PXE Specification). remote.0 determines if system is has adequate resources and downloads remote.1 (in specs to in extended memory at e98000h).  Monolithic pxeboot is about 200K, with pxe_http above 240Kb. I don’t know if 32K size is strict requirement or not, pxelinux for example is about 12Kb.

Well, “NFS loader works” I’ve thought and then reduced binary size by removing all pxe_http testing code and code related to support filesystems such as dos. It gave me nearely same size as usual for pxeboot. So, now http loaded kernel boots on my home machine.

While working with DHCP client, I’ve found that I’m doing same work, that was performed in bootp() call in libstand. I was not giving much attention to it, cause my code was working, but after binary reducing thought, that it’s good idea to use already available function. This function mainly depends on udpread/udpsend functions, that were earlier situated in libi386/pxe.c but commented by me while rewriting it’s code a little bit (commented cause it was using pxe_call and etc, that was removed by me form this file).

It was not big deal to rewrite udpread/udpwrite to use pxe_http functions to read/send data, but for this purpose was done “default socket” mechanism for UDP. It’s done with aim not use sockets while calling udpread/udpsend, cause code, which uses them, don’t know about pxe_http sockets. To be more accurate, sending of udp packets doesn’t require socket, it’s enough to call pxe_udp_send(), but incoming packets are coming through filters mechanism and are delivered to appropriate socket buffer. Previously, all data that was filtered out – was going to trash bin, now it goes to “default socket”. and it’s possible to read from it as usual, using pxe_udp_read() function.

So, bootp() began working good (except moment with updating of gateway/nameserver information, that I’ll fix in next few days). The main problem with it was, that bootp() may work only after opening of device, cause after that NIC MAC is known by code, that generates BOOTP packet. Normally, pxe_dhcp_query() – which is now may be wrapper for bootp() – is started from pxe_core_init(), and already knows own MAC. Well, I’ve returned pxe_dhcp_query() to pxe_open() in case if not own DHCP client used (here was bootp() originally). And all began work correctly.

NFS code also uses udpread/udpsend functions, so it was rather logical to try NFS loader. It also works. It’s started working without any doubts. so it’ uninteresting to tell anything about it.

This made me possible to make conclusion that compatibility with old working PXE related code in libi386 is big enough.

So, formally main goal of project is achieved. But there is more work to do:

1. load image of RAM drive and mount root to it, so booting process via http will be fully performed. Now it stopson stage of mounting of root file system.

2. Understand where is problem with memory when I’ve used bigger buffers with big pxeboot binary. It’s rather interesting moment, cause I’ve no idea what might cause rebooting of prefectly loaded kernel in case of bigger buffer size for sockets earlier.

3. Test all and update some code details (well, one of them is that root path is not used by http loader now).

4. write finally whole documentation related to project. I’ve started to write it few times, but it wasn’t finished yet.

5. choose notebook to buy.

These are my tasks for next weeks.

Kernel loaded

July 25th, 2007 by taleks

httpfs now works. It uses pxe_http code to open keep-alive connections to http-server. Also it ressurects it to life if connection is dead due server’s settings (e.g. timeout or exceeded max requests per connection). Need to think how to make requests optimal. Loader uses 4K buffers at request, may be I’ll add some sort of caching mechanism for httpfs to reduce count of requests to server.

So, it’s possible to use common commands of BTX such as ‘more’, ‘load’, ‘help’. I’ve played a lot with them while testing of http code, rather boring process. But after that one problem with interrupt handling in pxe_core was located. It seems, that end of interrupt is handled not correctly, so interrupt not occurs from time to time till some unrelated top pxe_core code finishes interrupt correctly. Problem is temporarily fixed by ignoring of __pxe_isr_occured flag and checking of available incoming frames in pxe_core_recv_packets() despite of value of this flag. Meanwhile TCP code was simplified by rewrtting it in pxe_tcp module, reducing code lines and function count. Testing showed it works identically to previous code, except one silly misprint, that was fixed in new variant of code.

Anyway, it’s possible to load modules, kernel and etc. It’s interesting that loader tries to load gzipped variants of files first, so it performs four requests to server. E.g. we need “loader.rc”, theese files will be requested: “loader.rc.gz.split”, “loader.rc.gz”, “loader.rc.split”, “loader.rc”. It was unexpected and exceeded limits of filters and connections tables, cause connections felt to TIME_WAIT state (keep-alived connection were actively closed by client. Need to try in nearest future sending of “Connection: close” field in header to make passive connection closing at closing of file), no new connections were possible to establish. Well, I’ve added opportunity to free connections structures in this state for establishing of new connections. It’s violates RFC, but helps keep relatively small amount of structures. If PXE_MAX_CONNECTIONS is set to big enough number, then TCP code behaviour is as expected in RFC.

After changing loader.rc to something very simple, my home made kernel was loaded and ‘lsmod’ showed a number of modules. But ‘boot’ command failed (hang or panic depending on which virtual machine and real machine were used). It was frustrating – result is practically visible, but yet goal not achieved. I’ve tried loading of RAM-drives, but also failed.

In bad mood of me normal loader.rc was started. The first surprise is that on VMWare machine I’m see normal booting process till time of mounting of root file system (which is unknown, cause not set by environment parameters in loader and /etc/fstab is empty in Apache’s DocumentRoot directory). Second surprise that another virtual machine hangs during loading of support.4th (according to Apache logs it’s read not completely, about 70-80%) and my real machine just reboots while reading same file. Rather suspicous :)
So next days I’ll be solving quest for finding why loading of support.4th, which have no nothing criminal in it, may cause such problems and exploring issue with lost somewhere interrupt.

Something looking like result

July 14th, 2007 by taleks

From the last post there were big steps ahead.

Second variant of TCP code was good enough and I’ve even got result ‘tes’ (3 letters of word ‘test’) on server side while testing. After some testing and improvments, whole buffer was transferred well, but there were no resending queue. It appeared in third variant.

For that purposes was needed a small and cute packet allocation manager (receive buffer is still uses common cyclic buffer routines, but send data buffer is used by packet allocation manager). Effective memory allocation in case of variable size of packets is rather difficult task, so I was implementing other task – allocating of fixed size packets. Simple enough to implement it in short period of days. All packets were divided in two groups – “small” packets and “big” packets. Small packets are used for system messages (SYN, FIN, ACK without data, RST), big – are for data transmitting.

How it works? Buffer space is divided in chunks with size equal to maximum size of “small” packet. It’s now 64 bytes (this is enough for IP, TCP header, TCP options and segment description). Every chunk contains TCP related segment description (resending time and other). Sequential chunks are gathered together in blocks. Block is used for allocating data for “big” packets and it’s size 512 bytes. So, memory manager just allocates blocks or chunks in place. Block may be exclusively be owned by one big segment or by count of small segments. Amazing, that it worked from the beginning without signifcant errors. Segment resending function uses segment description, so it’s possible to send segments allocated using other manager or just statically allocated (used in sending RST packets when there are no connection, thus there are no socket and thus there are no buffer and space to allocate chunks)

So memory manager was ready and resending become pretty simple – just check resend time of segment and resend it. And I’ve started simple testing of code by sending blocks of generated sequences with identical initial generators on both sides of connection. And it worked perfectly till summar transferred data to one side was less then buffersize. Bigger amounts of data make code crazy, guard checks of memory allocator become very talkative and loved easily to fell to panic. Few days of testing located problem part of code in buffers read/write code in some cases. After standalone tesing module for buffers all become correct and data started transferring well. First test was in trabsferring 8 MBytes of data (that exceeds defsault buffer size 32K). Good new was that data is going correctly: per byte checking showed no errors. Bad new was that speed was about 5-6Kbytes/s at FastEthernet. Really annoying to wait all this 8Mb to be transferred.

The reason was simple – after filling receiving buffer and getting small window size (less then default mss) server side was waiting about second or two before asking client side, is any free space now. So, one fix in pxe_recv(), which started checking connection by sending ACK every 100 ms if receving buffer have much free space, saved the Earth, pardon, project. Speed become 500K-1000K. Well, still is not best, but better than was, next speed up may be done by sending/recv’ing bigger blocks (in test it was byte by byte in order to check sequences are correct). Now there is another problem :) client code is very nasty if server is not sending anything, client bombs server with ACKs about big window ready to work. I need to think how to make client behavior more correct. At this point 80Mb of data were transferred correctly and relatively fast, so test was over with verdict – passed.

It was so good, that even warned me a little bit. Time showed I was right, not all was so good. To be more accurate – connection breaking was handled incorrectly, when server initiated breaking (passive closing from client view on this situation). Some updates in tcp_disconnect() helped to solve problem and LAST_ACK, CLOSE_WAIT are now also handled in right way. At least I think so and tests prove it. This changing in disconnect function give also speed up bonus in active connection closing.

But anyway, next version of TCP implementation is already in thoughts to simplify pxe_tcp module and reduce function count. May be some other changes will be done. E.g. I’m trying to reduce sending of ACK packets by using incremental ACK (now every incoming segment is confirmed). This already is performed in checking of “same” packets: every packet that is just copy of current, placed in queue, with older sequence number to ACK is removed from resending queue. It helps a little bit, but solution is still not good enough.

Meanwhile I was doing http client, the only function of which was to fetch chosen file (I was testing on html-page) from web-server. There was not anything special, except that I was sent again to read style(9) and implemented simple snprintf() in libstand by modyfing PCHAR() macro. http receiving code was working, so next function implemented partial requesting of data. It’s rather simple, just adding of other field in header (assuming server supports such feature). Main task to do here in future – don’t establish connection for every reading of part of file. Now it established at start of reading and closed after that.

Next step, I forget about – filesystem. Well, for reasons I don’t know, I was thinking pxe.c is all that needed to change and files become getting itself from web server (by the way, www-server option was added to DHCP-client to get it ip). But it’s rather logical to separate filesystem and device. So, I was needed – make stub filesystem code. Using already available code it was simple – just call pxe_http module functions at needed time.

So, next days I’ll continue checking and correcting of existing code (also, once more style updates…), but main task will be tie together filesystem, pxenet device, http code and see what will be in result of this mix.

First TCP steps.

June 21st, 2007 by taleks

From time of last post there were big enough changes in code.

First of all, await functions support was implemented, so pxe_core_recv_packets() function now is hidden from user code even deeper.

Await functions are functions that are called back by pxe_await()  during simple waiting mechanism. It seems in some modules it made code smaller in lines, but added wait_data structures, which stores current wait related variables. As for TCP module, it seems await function is not useful, but to be similar to all other modules – waiting during handshaking is done using tcp_await()

pxe_await() provides event, current try and wait related data when executes await callback function. So, all await functions are similar and are processing next try, new packets, start of try, end of try events.

DNS client was rewritten a little bit to use await functions mechanism, DHCP client was written to support it from the first lines of code.

After different issues with DHCP protocol simple client was produced to get nameserver, gateway and other useful as well as useless information from DHCP server. May be I need also to add project related DHCP-option, which will specify from which server to download files.

After implementing of major UDP services needed by project, finally TCP module got its first implementation.

As earlier in other protocols, main function for packet processing is callback named simply pxe_tcp_callback(). Main difference from other protocols, especialy UDP – that sockets and filters here firstly used separately and actual work is done in state handlers, not in main callback from start till end.

State handler functions registered now in compile time (and I believe never will be registered in run time, although it’s possible) in state_functions array and each function serves for one TCP connection state handling, if it’s NULL, state is ignored.

Connection is binded more to filter, than to socket structure. This is done to handle correctly TIME_WAIT state, when there is no actually any socket, but any lost packets must be got and forgot.

After testing in soft conditions of LAN it seems active establishing and active closing of connection is done correctly.

Next things to do:

1. resending of packets after timeouts. Adding max segment size option to tcp packet.

2. data send/recv using TCP sockets. Adding pxe_push() function to push data.

3. queueing of received not in correct sequential order packets. Now they are just dropped. It may be useful to move core_packets structure to TCP module for this purpose.

DNS client and etc.

June 9th, 2007 by taleks

Well, first version of DNS client was working about a week ago, but it had one problem: some of domain names were resolved, others were not. In fact, even packets were not returned in reply to sended requests. I’ve checked request making function and found no problems, added verbosity of debug information and found no problems still. After some thoughts, I’ve decided that only lower levels of code may produce such fault. And after some hunting for bugs, one fat error was found at UDP checksum calculation. UDP checksum is calculated for pseudo IP header, UDP header and data. I was calculating this sum separately (for pseudo header and for rest) and after that adding them together. On this stage there was error and routers, which check UDP checksums, just were dropping this packets. When local DNS server was able to resolve cached name (requested by other DNS clients in LAN) it was answering, cause it was not checking sums.

DNS clients (and currently developing DHCP client) are implemented using UDP sockets. For sockets I’ve added cyclic buffers and routines to work with them. UDP callback functions stores datagrams (which passed filters) to socket recv_buffer, heading structure with source address information that is used by pxe_recvfrom(). There is no ability to know if datagram is truncated now. if user recv’ed less then total size of datagram – rest of it is dropped.

Added after this pxe_bind() function required using of socket state. Now it’s not only free/used, but binded/connected/established. Binded socket has only one part of filter set strictly, connected – both source and destination related parts, established is unused, but will be used later with TCP.

DNS client is used in pxe_gethostbyname() function which returns ip4 addr by domain name. Need testing of CNAME handling, if there is no A record in additional or answer section (in this case request resended with CNAME to resolve. In my tests there was no DNS server, which returned somedomain CNAME canonical_domain without leading canonical _domain A ip.ip.ip.ip.). Resolved names are not cached, I don’t think it’s needed for this project. Meanwhile testing, i’ve found funny thing: www.google.ru is CNAME to www.google.com, but google.ru is just A to set of ip’s (by the way DNS client takes first appropriate record as ip resolved for given name). So just three letters and you are logically in other network.

UDP sockets seems working (I’ll test it later on NFS client in pxe.c to be sure there are no bugs). pxe_sendto() moves socket to connected state if it was not binded before (it’s used in DNS client, may be it’s better to change behaviour of this function)

Next to implement:

  • universal waiting mechanism for packets to avoid repeating of cycles in every module. It will be similar to Etherboot or to the uIP await functions.
  • getting default gateway/nameserver from DHCP server. Although, it’s possible to change them manually (‘pxe ns’ and ‘pxe route’ commands in testing module) it must be get by default from DHCP server (also, in DHCP packet may be returned host from which perform downloading of files via http. this need adding user option to packet). For some reasons cached DHCPACK packet data in PXE is filled with zeroes (during downloading of NBP, TFTP has this information, it shows it, but returned by PXE API call data contains no information in options section. May be I need check legacy BOOTP section if it hides needed info instead of filename and servername)
  • start implementing TCP recieve/send packets function.
  • add raw_handlers. Currently, if protocol callback function does not think packet useful for itself, function drops it. raw_handler for protocol will be callback, which will get this packets. it’ll be set by users. It’ll no change currently working code except of return values of callback functions.

Some thoughts about sockets

May 26th, 2007 by taleks

I’m deciding, what is more important: syntax compatibility with usual sockets implementation (mimic function declarations and function appearing in user code order as much as possible ) or basic compatiblility (module behaviour is similar, but some functions named differently and function sequences to establish connection may be another) is more than enough. May be it’s one task, but views from two sides. This day I’m thinking first variant is more appropriate, but it limits some project functionality available with current understanding of pxe_filter.
All incoming packets flows from hidden (may be not hidden, anyway userbased networking needs receive cycle) from user pxe_core_recv_packets() function to registered by IP based protocol callback function (e.g. for UDP – pxe_udp_callback(), installed by pxe_udp_init() call at pxe_core startup). Callback function must dispatch packet data to one of sockets, if any inited.

For that purpose Ive create module, named pxe_filter. Every socket at eatablishing connection time (or at time of equivalent of listen() call, or just at socket creating time) installs filter to filter table. Filter conatains destination/source ip/port with masks for every parameter and reference to socket, for which installed filter. So, protocol handler function starts pxe_filter_check() with extracted from packet values of ports, ips and protocol type. Filter check function sequentialy checks filter entries from start to end entry, and returns when (if) first satisfactory entry found (well, similar to firewall rules). If filter entry found, socket is extracted from pxe_filter_entry structure and socket member of structure is valid – data dispatched to the socket buffer, if there is free space in it.

What benefits it gives:

  • it’s simple. And it adds some functionality of primitive firewall (which may be not interesting in this project).
  • not connection oriented code may listen only that ip in whcih it interested. Usually check from which ip datagram received is performed after recvfrom () or recvmsg() by one of returned values. But in case of filters – there would no necessity in this. Filter may limit from which ip to get packets. I think, it’s useful, e.g. for DNS resolver or any other non broadcast UDP packets transmiting, when it’s known from which ip data expected. If mimic absolutely sockets function syntax, this functionality may be lost.
  • listen() call may use filter to “fork” sockets, creating socket with more strict filter higher (earlier in checking order) in filter table. So all accepted connections may be assumed as child to base socket. And it’ll help to check if there are more connections to accept and after closing of base socket – which connections also to kill, if needed. Well, my project is more client oriented, than server, but anyway good opportunity.

For me, it seems not bad idea at all, so I’ve started doing it in this way.

PS:

can someone say how to fix SpamKarma 2, which is installed incorrectly? all links from manage page links to freebsdish.org/wp-admin/… but not to freebsdish.org/taleks/wp-admin/… it’s a little bit annoying to enter manually

…/taleks/wp-admin/options-general.php?page=spamkarma2&

sk2_section=spam&sql_rows_per_page=20&

sql_skip_rows=0&sql_score_threshold=-100 to check if some comment is mistaken with cialis ads.

Ping works

May 23rd, 2007 by taleks

Somehow PXE code works, and in nearest future only C and TCP/IP related stuff expected, no assembler and PXE related problems. Last issue (code worked only in one available to me implementation of UNDI API ) was solved by modifying most used function pxe_core_call(). For some reasons, it returns incorrect value in ax register, however code works correctly. So, ax register now ignored till understanding why it’s contents is corrupted.

IP imlementation was made really fast, but checksums of IP and ICMP headers provided some nervous hours: ICMP reply (from PXE client) packets were noticed in tcpdump, but ping itself ignored this packets. Problem was in incorrect icmp checksum (first time tcpdump wasn’t saying me that there is problem with icmp checksum, but after some changes in checksum calculating it began warn. I guess, why it was not warning earlier, when checksum was calculated only for icmp header without transmitted data? May be it was a miracle and this bytes were filled by zeroes, so checksum was independent from transmitted data) .

The second problem that was expected – routing of packets to gateways. For this purpose, some routing related functions were implemented, and now it’s possible perform ping of death from PXE booted workstations :) Routing functions too primitive in my project, I need do something with them. By default, at start (pxe_ip_route_init()) two routes are added: default (it must be got from PXE cached packets, but there are only zeroes in place of gip member, so I set my home gateway 192.168.0.1 as default gateway. After implementation of some DHCP/BOOTP functions, it’ll be possible to get gateway automatically). pxenet0 (this route for hosts in local network, in fact – it’s not gateway. It’s named as interface in original pxeboot). IP related code uses first route, which network is equal to destination network of ip packet.

For testing purposes I’ve added to pxeboot command pxe, which allows to test project subsystems:

1. pxe arp ip.addr – sends ARP prequests for provided ip. Returns MAC address of desired ip, if host is upped.

2. pxe ping ip.addr – sends 5 pings with 32 bytes to provided ip.

3. pxe route – change route table. e.g. ‘pxe route add default 10.0.0.1′ will set 10.0.0.1 as default gateway. As you see there is no mask, it’s calculated from ip address class (no CIDR, so…). ‘pxe route print’ – shows current routing table.

4. pxe await – starts infinite cycle ofchecking received packets. May be used for client pinging purposes or serving client as server. There is no exit from this test module.

Well, in fact I need somebody to test modified pxeboot in PXE environment to be sure code is compatible with most existing PXE implementations (I’ve tested all available to me implementations and it worked, but who knows…). If somebody, may find a time and test – help will be greatly appreciated.

Here is tar.bz2 of modified version of pxeboot. It’s needed to up DHCP server with appropriate  settings  and TFTP service.

First result

May 20th, 2007 by taleks

Well, first of all, about pxe.h. It has error in structure naming t_PXENV_UNDI_INITALIZE, it took some time to find, which letter is missed. Other issues with this fie: not armoured header (which disturbed me while compiling, cause included twice), PXENV_UNDI_GET_STATE related structure and definitions are missing. I’ve updated it only locally, but may be it’ll be better to make perforce mapping later.

After successfull complilation of libpxe_http.a, I’ve thought all is good, however linker thought differently. There was “R_386_16 relocation truncated” error. Now I know, it was gas ‘feature’, it’s rather problematic to write mixed 16 and 32 bit code with it, but first time I was in confusion. Now pxe_isr.S uses manualy rewritten segment:offset adresses, based on calculating sement and offset to data_start. I’ve thought about using code segment as data segment (in fact used data is placed in code segment), but first try gived me bad results, so for first time it will be somehow ugly, but working code

ARP related code was already near working state, so main problem was how to test it. I’ve modified pxe.c (implements pxenet device and NFS (via UDP) loading of kernel), just to start test of arp (this test sends broadcast request who-is provided ip and receives replies, updating internal arp_table structure). And after some trys understood, that pxe_core_receive() code is doing nothing, cause UNDI returns pointer to buffer, to which there is no access from userspace (starts at 0xa000, buffer is in lower addresses). So, __mem_copy() in pxe_isr.S was born with destiny to copy data in vm86 mode, and data_buffer array finally came in handy as place to copy to. There was also bug in receiving cycle, so I’ve made debug macro and started printing out to screen everything useful and useless. And have found, that receiving buffer includes media header and transmit (at least, for known by UNDI protocols) doesn’t include ethernet header.

Meanwhile, command ‘arp’ was added to loader (just to check, if I understand right how to add cmmands there), it performs arp test (but to say true, it not works, cause earlier started by autoboot pxe related code, and thus other test, from pxe.c function pxe_open() which shutdowns undi after all).

After fixing of some bugs and rebuilding – arp test began working in virtual machine (I am omitting tonnes of messages ‘unknown hardware address format’ in dmesg :) and some stupid errors with broadcast address for ethernet). It sends broadcast message for my home computer in localnet and receives reply, caches it and finishes. Well, I was such happy, that even rebooted desktop trying to perform test on one of it’s NICs (internal Realtek 8168; there is also 3Com 3C905C-TX, but I’ve not tested on it yet). Well, as always, reality was cruel. All pxe_core_transmit() calls were unsuccessfull, I’ve tryed other virtual machine (with different implementation of PXE) and found the same result.

I think, problem is incorrect sequence of initialization of UNDI services in pxe_core_init(). PXE spec says, that on remote.0 stage (started pxeldr) it’s possible to stop everything (STOP_BASE. UNLOAD_STACK, STOP_UNDI) and start remote.2 (the same executable in my case) with initialization UNDI again, but after that even first virtual machine rejected my code (mmm… in fact I’ve done also UNDI_CLOSE, UNDI_CLEANUP, UNDI_SHUTDOWN). there is also two mysteries – GET_STATE function always returns ‘failed’ result, but it seems, returned UNDI state value is not such incorrect. And some transmit calls fails with Status = 0 (which is something like ‘not any error’ status). It’s strange (I guess if I understand right – call failed, if ax register != 0).

So main problem for next few days – to find correct initialization sequence of calls, if problem is in that. And when problem will be solved – start first testing and implementing of ip protocol.