Prospects and Dangers of Large Scale Social Data Collection

A new site about Very Large Scale Social Data Collection  based on Recursive Exhaustion only hints at the opportunities and dangers presented by methods for collecting enormous amounts of information about individuals, corporations, institutions and geographic areas.

Opportunities include highly focused advertising far beyond anything used today and the ability to effectively conduct business with people by exploiting their newly visible strengths and weaknesses.  Dangers include the prospect of being manipulated by people exploiting your own strengths and weaknesses as revealed by the data.

More sinister dangers include people who will try to manipulate or exploit you using blackmail and intimidation based on this data.  On the plus side, the data can also be used to reveal the people trying to do this.

In short, the ability to collect massive amounts of information about people is likely to change the social landscape enormously.  The same methods can also be used to gather information on corporations and other institutions.  A clever businessman could use that kind of information to maneuver among these corporate entities and carve out a better niche.

So as well as changing the social landscape enormously, the use of this kind of data could change the corporate landscape beyond recognition.  That is a wave which one could ride, but risks being swamped by.

Posted in Uncategorized | Leave a comment

New Decision Estimation Site

Decision making and estimation are central to the operation of a successful business.  They are at least as important in the creation of a new enterprise.  Though intended for many other applications of social technology, a new site has been created to explain new methods of decision making and estimation, and to promote software development based on those methods.  Please visit for more information.

Posted in Uncategorized | Leave a comment

Motives: For Love or Money

As I see it, there are two main reasons why one might be interested in advanced social technology — love and money. By ‘love’ I do mean something more general, having written of many aspects of a social environment, spousal love, friendship, career, education and place of residence, all of which have much to do with quality of life and all of which may be good reasons for loving the life you lead. Regardless of any entrepreneurial or investment interests, these remain a strong motive for me to seek to promote advanced social technology. You can hardly deny their attraction.

Then there is the question of money. Let me leave my own modest ambitions out of this — I’ll explain later or elsewhere. Others may have a great interest in making money, which many consider the strongest motivating force of all. Whether one shares that belief or shares a strong desire to make money, one should appreciate that the advanced social technology we want will come sooner if some people can make money with it.

There are various ways in which a belief in social technology could lead to financial success, perhaps even overwhelming financial success. One level would be the “garage” entrepreneur, the technical person with ambitions, who develops new technology from home. Several people have gotten extremely wealthy doing so, those who had the insight, skills and dedication.

At the other end of the scale are those would make money simply by investing in businesses which offerred hi-tech social capabilities. If Yahoo decided to support and incorporate advanced social technology all the way, while Facebook decided to continue with its medieval technology and Google rejected it completely, then buying Yahoo stock while selling Facebook and Google short could bring in a lot of money.

Somewhere between these extremes is the serious entrepreneur, someone with money to start up and invest in a new high-tech social venture. This person would contact good technical people, provide them incentives and support them with management and support staft instructed to interfere as little as possible with the experts, then seek for ways to either market the product or the enterprise as a whole.

For the record, I am most interested in making this all happen, however it happens, as soon as possible. While not immune to attractions of money, that is a long way from my own higher priorities. As I hope I have made clear, I will do whatever is necessary to make this happen soon. In the past I have probably emphasized too much my own role in this. It is not clear if I am an asset or a handicap — does anyone really want a tech fanatic who reads too much, thinks to much and writes too much? What I would most like is a chance to educate and work with others. On the other hand, finding the right people to teach or work with is not easy. Advanced social technology could make it easy, so I am very interested in supporting in any way I can those people who for whatever motives share my desire to get all this going as soon as possible. — dpw

Posted in Uncategorized | Tagged , , | Leave a comment

Design and Implementation Sketches

Very brief summaries of some of the technology.

— Data Mining, some methods very sophisticated, e.g. using neural networks

— Data collection, massage and correction, using various linearization methods, factor analysis, and its converse, synthesis, rotating the data as factorized into orthogonal components into frameworks suitable for application, not for scientific study, finding appropriate de-linearizations to undo the transformations which made the principal components or factor analysis into a linear algebra problem. Correction of data throughout this process using missing data algorithms, and considering answers to apparently redundant questions, some more prone to response-set effects.

— question generation by factor analysis on rows (questions) rather than columns (answers) when doing questionnaire data analysis, followed by searches through a large database of existing questions with some known response stats, to select questions which will get more and more of the information wanted, through an ongoing dialogue with the user — using a sharp instrument instead of a blunt one

— application of graph theory (or network analysis as sometimes — incorrectly — called) to do analysis of vertices (e.g. people) and edges (e.g. communcations channels or interpersonal relationships), with the aim of synthesizing more useful or “better” graphs, complete graphs, biconnected graphs, transitive graphs and perhaps especially, bipartite graphs. The most computation difficult problems arise in attempting to solve such problems, such as the notorious travelling salesman problem, which actually does arise, as e.g. when attempting to link stable spousal relationships through friendship relationships (man-woman -> woman-man -> man-woman -> woman-man, etc., forming a single closed loop where each person has exactly one spouse and one best friend.

— differentiating individual characteristics and the interpersonal relationships they imply with respect to time. Not just is this couple compatible today, but considering how each person is changing, will they grow closer or further apart, more or less compatible. Second derivative: is this changing over time. Perhaps they are indeed becoming less compatible with time, but rate of change is slowing, or perhaps reversing. Perhaps soon they will stop growing less compatible with time and may start growing more compatible with time.

— Group theoretic methods. To solve a Rubic’s cube, simple rotations to line up two cubelets will have the side effects of misaligning others. To solve the cube one must develop elaborate sequences of rotations, based on subgroups, which will do the desired move only. Similarly to optimize your own social network (never ever think of anyone like the government imposing some such optimization) — to work in your nearby social environment only, various sequences of changes can have the effect of accomplishing the desired simpler changes without the side effects of doing them directly.

— global social network optimizing, again by suggestion only, using simulated annealing algorithms where by the structure as a whole is treated as either a glassy or crystalline structure, as appropriate, with defects. These defects can be thought of as strained edges or as missing edges which should be connected in a graph (network). Relaxation methods can be used to model the possible annealing of this structure to remove these defects, and the resulting changes turned into suggestions for the user’s use, with tools to be provided.

Well, these are just sketches, absurdly brief, just hints of what can be done and how. I can talk endlessly about such things, and would be glad to do so for the possible amusement of selected technical experts from serious business interests, or well, just for mutual satisfaction, with Michael Ferguson’s erudite and refined polymaths. — dpw

Posted in Uncategorized | Leave a comment

A Google Model

As I have said to many people I worked with when doing requirements analysis, “Stop thinking about what might work, think about what you want.” Scenarios, dreams first, then let the designers brainstorm about possible implementations. I posted here a scenario based on Facebook friend suggestions. Here is another, based on Google searches.

Google already retains some information about users, especially if one signs up for the quite useful iGoogle, a good way of making yourself a home page. Let us be less concerned about how much information Google (or a similar search engine) might retain about you, though certainly being concerned about privacy and security.

Suppose that you are conducting searches on Google, as I do many times a day. The best response to your query may not be a webpage or other document, it might be a person. “Yes, read this page, but best of all contact this person”. That is a bit too aggressive and specific. What you probably want is not to add your request to the inbox of some busy professor who receives hundreds of such requests a day. Instead, you should be directed towards someone like yourself, seeking similar information, but probably coming at it from a different direction.

There are several problems with this. Many people may use the same machine, utterly confusing user profiles. I have written of using a single button click by which the user can identify him or her self from amongst other users in the home or business, info which might quickly expire if the machine is idle. Another problem is the diverse interests a single person might have. What does my interest in collecting pictures of pipe organs for my screensaver have to do with my interest in social survey data? Proabably little. But search pattern can help with the basic requirements which I would impose on such a system.

It should be global, or at least a default, so unless a person opts out of receiving such suggestions, he or she should be directed towards suitable people instead of just receiving lists of web pages. It should be a default in the other direction as well, but easy to opt out of. Unless you specifically say so, people might be directed towards you. But that in turn raises problems.

What do you want? E-mails from suitable people? Perhaps. But perhaps not. As with the Facebook example, I think a two-step process should be initiated. First, when you search for something, (e.g.) Google should point out the existence of people looking for the same or similar things, or the existence of people who often look up the same things you do. Second, each person should be told of the others’ settings, preferences, or extrapolated preferences. Are you willing to accept contacts from others researching this topic? Are you interested in contacting others researching this contact. If both conditions apply, then the search engine should produce a list of summary descriptions to click on.

If, for example, I am often looking for free or cheap 64-bit software, and I am often looking for social survey data, then the search engine could respond with an anonymized list of people to contact — implying or explicitly saying “You have expressed a preference for being informed of suitable people to contact, as have the following individuals. Click on those of interest”. If you click on one person, the software should respond with something like “this person has expressed a willingness to receive e-mails from suitable people”, or, “this person does not want anonymous e-mails, even from those researching similar topics, but will be notified of your request next time he or she makes a search relating to this topic”. Once connected anonymously people could be left to establish more direct contacts on their own, or the software could facilitate this.

As in the Facebook Fantasy, this is a scenario and only the barest hint at what might someday be written in a software requirements document. But that is the right thing to do, say what you want first, without asking if it is possible or how it might be done.

What do you want, exactly? It can be very hard to spell out. Especially hard to make it symmetric or bipartite. You may want a connection with a beautiful movie star after filling out some questionnaire. But that person probably does not want friend requests from all 2 billion of her adoring fans. She wants the few most appropriate, only. So, you spell out a best match scenario. In the Facebook and Google fantasies great care must be taken to make sure the scenario is not one sided. The golden rule (in one form or another) must apply. For example “Do not expect connections from those who would not wish to connect with thee.”

As in the Facebook fantasy, there is room for Google or another search engine to undertake this entirely on their own. But here there is much more room for a small startup business to be extraordinarily successful. A new seach engine can be established without much difficulty because spidering the web is not hard. The mere possibility of being able to find a person as the best response to an ordinary query for information on a topic is likely to attract a user base. With even a relatively small user base some quite useable info-query to person matches could be made. An entrepreneur willing to take the initiative here could not just produce something Google might be willing to buy, but something which could replace Google at the top of the heap.

By the way, do not neglect matches in the other direction. As well as query-to-person matching, I have also written about person-to-query matching, by which people’s profiles would be used to suggest topics for their research, even though that research may be simply for popular forms of entertainment. Related to this query-to-person Google type matching idea is one of project-to-project matching, in which all of the queries made by members of a project are matched to all of the queries made by members of another project, with the aim of binding together projects for mutual-aid relationships.

I encourage people to come up with other technological fantasies, and have in fact written a book with that very title, “Technological Fantasies”, available from my page — it includes a few of the ideas just mentioned. — dpw

Posted in Uncategorized | Tagged , , | Leave a comment

A Facebook Model

Elsewhere I have repeatedly argued about the necessity for a large pool size, also pointing out the difficulties of getting more than a puddle. Let us think about one significant possibility, selling (metaphorically or literally) the idea to Facebook, so one could take advantage of their large pool of users. A note for entrepreneurs, some services like YouTube grew from the work of one man, who sold it to Google for over a billion dollars. I’ll discuss this again, elsewhere. For the moment, let us just take a look at how this would work, using Facebook.

Facebook’s friend suggestions are almost worthless, and, I would say, potentially harmful. It’s easier to see why they are of so little value — I do not need or want a suggestion that I be friends with my second cousin’s babysitter simply because we happen to have a few “friends” in common. Having friends in common is not a good indicator. So what would be a good indicator?

Ideally, having already signed up for Facebook, making some use of it, as do millions of others, you should be able to make some one-in-a-million friends, the best out of a vast number of candidates. How would this appear to you, the user?

I suspect the first step would be about anonymous people, anonymous but well described to you. Occasionally you should receive a system generated message saying that some suitable candidate exists, that someone perhaps thousands of miles away, entirely unknown to you, your friends, or even your friends of friends is nevertheless an almost perfect friend for you. A single click would let you confirm that you would accept such a person, someone matching that description, as a friendship candidate. The corresponding suggestion will have been sent to your potential friend, who would thus have the opportunity to welcome you as such a candidate. If you both agree, if you like the descriptions, then specific suggestions would follow — not suggestions for anonymous people, but more like ordinary Facebook friend requests for a person whose profile you could look at. A two step process, but perhaps requiring no more than two mouse clicks to link you to a one-in-a-million friend.

This is just a scenario, but the beginning of serious analysis often begins just this way. It is probably best to ignore for now the burning questions of possibility, how it could be implemented, how it would work. The important first step in making this happen would be to say “Yes, this is something I want — please make it happen.”

And yes, this is something I want. I hope someone does make this happen. I would be astonished to learn that none of the intelligent people working for Facebook could implement something like this, but I wonder if they would. If not, there are other social networks out their with large enough memberships. One might agree to do it, and could figure out how. On the other hand I think there are many chances for an entrepreneur to take the initiative, getting something that could be used on a social networks members, providing its members a highly desirable service, attracting new members to the social network — and bringing rewards to both the people who first took the initiative and those who run the network.

I have worked on this for many many years and could tell people how to set this up, though my own involvement may be limited. On the other hand, I couldn’t resist actually starting a software development project of my own, which is not yet at the coding stage (as is should not be). Aside from fantasizing about getting access to the huge Facebook pool of users, I most often dream of my feeble efforts turning into a big open-source software development project, on the scale of GNU, to bring advanced social technology out in the open where everyone could use it. But that does not mean ignoring commercial possibilities. I will explore some elsewhere, in a scenario replacing the Facebook one with a very different Google one. — dpw

Posted in Uncategorized | Leave a comment

What Counts as Advanced Social Technology

Well, not Facebook, that is certain. Another discussion can say why not. Advanced Social Technology would make use of a lot of mathematics, like any advanced technology. Exactly how it works would probably be a trade secret, for fear of imitation, as with Google, which actually does use quite a bit of mathematics. Advanced Social Technology would use a lot of the methods developed in the Social Sciences, for data collection and correction, the massaging of data, along with data mining. For social networking, the sophisticated computational methods of graph theory, (a technical term for what the public might call network analysis) would be employed, using existing sophisticated algorithms and probably developing more. The difference between advanced social technology and the primitive kind represented by
Facebook would be like the difference between the research and development of modern advanced medicines compared with the primitive patent medicines of over a century ago.

Posted in Uncategorized | Tagged , | Leave a comment